task e2e-llm-inference-service has failed: "step-fail-if-needed" exited with code 1: Error [get-kubeconfig] Found kubeconfig secret: cluster-g6fqs-admin-kubeconfig [get-kubeconfig] Wrote kubeconfig to /credentials/cluster-g6fqs-kubeconfig [get-kubeconfig] Found admin password secret: cluster-g6fqs-admin-password [get-kubeconfig] Retrieved username [get-kubeconfig] Wrote password to /credentials/cluster-g6fqs-password [get-kubeconfig] API Server URL: https://afd3bdff287004064be9628906775d0c-001fd88cd0fa6e57.elb.us-east-1.amazonaws.com:6443 [get-kubeconfig] Console URL: https://console-openshift-console.apps.007a4926-ebb1-4604-8489-b69a2896cd17.prod.konfluxeaas.com [clone-repo] auto-sync/master-to-release-v0.17 [clone-repo] https://github.com/opendatahub-io/kserve [clone-repo] Cloning into '/workspace/source'... [clone-repo] Updating files: 46% (1527/3267) Updating files: 47% (1536/3267) Updating files: 48% (1569/3267) Updating files: 49% (1601/3267) Updating files: 50% (1634/3267) Updating files: 51% (1667/3267) Updating files: 52% (1699/3267) Updating files: 53% (1732/3267) Updating files: 54% (1765/3267) Updating files: 55% (1797/3267) Updating files: 56% (1830/3267) Updating files: 57% (1863/3267) Updating files: 58% (1895/3267) Updating files: 59% (1928/3267) Updating files: 60% (1961/3267) Updating files: 61% (1993/3267) Updating files: 62% (2026/3267) Updating files: 63% (2059/3267) Updating files: 64% (2091/3267) Updating files: 65% (2124/3267) Updating files: 66% (2157/3267) Updating files: 67% (2189/3267) Updating files: 68% (2222/3267) Updating files: 69% (2255/3267) Updating files: 70% (2287/3267) Updating files: 71% (2320/3267) Updating files: 72% (2353/3267) Updating files: 73% (2385/3267) Updating files: 74% (2418/3267) Updating files: 75% (2451/3267) Updating files: 76% (2483/3267) Updating files: 77% (2516/3267) Updating files: 78% (2549/3267) Updating files: 79% (2581/3267) Updating files: 80% (2614/3267) Updating files: 81% (2647/3267) Updating files: 82% (2679/3267) Updating files: 83% (2712/3267) Updating files: 84% (2745/3267) Updating files: 85% (2777/3267) Updating files: 86% (2810/3267) Updating files: 87% (2843/3267) Updating files: 88% (2875/3267) Updating files: 89% (2908/3267) Updating files: 90% (2941/3267) Updating files: 91% (2973/3267) Updating files: 92% (3006/3267) Updating files: 93% (3039/3267) Updating files: 94% (3071/3267) Updating files: 95% (3104/3267) Updating files: 96% (3137/3267) Updating files: 97% (3169/3267) Updating files: 98% (3202/3267) Updating files: 99% (3235/3267) Updating files: 100% (3267/3267) Updating files: 100% (3267/3267), done. [e2e-llm-inference-service] + bash [e2e-llm-inference-service] + STATUS_FILE=/test-status/deploy-and-e2e-status [e2e-llm-inference-service] + echo failed [e2e-llm-inference-service] + COMPONENT_NAME=kserve-agent-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-agent-ci '.[$component_name].image' [e2e-llm-inference-service] + export KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:efaf0bd4be5e59e0fd5df072416f565ae616956e4a23f814ea31d63b5efa6d4b [e2e-llm-inference-service] + KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:efaf0bd4be5e59e0fd5df072416f565ae616956e4a23f814ea31d63b5efa6d4b [e2e-llm-inference-service] + COMPONENT_NAME=kserve-controller-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-controller-ci '.[$component_name].image' [e2e-llm-inference-service] + export KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:e84ef678972994166b9c11fc0192a03ca1e4fecf65c513bcc81ed8eba01106de [e2e-llm-inference-service] + KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:e84ef678972994166b9c11fc0192a03ca1e4fecf65c513bcc81ed8eba01106de [e2e-llm-inference-service] + COMPONENT_NAME=kserve-router-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-router-ci '.[$component_name].image' [e2e-llm-inference-service] + export KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:24a06dcfbbe7f0d25115c5a79181136502c6b1ca34878eb57638d7eb57f020b2 [e2e-llm-inference-service] + KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:24a06dcfbbe7f0d25115c5a79181136502c6b1ca34878eb57638d7eb57f020b2 [e2e-llm-inference-service] + COMPONENT_NAME=kserve-storage-initializer-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-storage-initializer-ci '.[$component_name].image' [e2e-llm-inference-service] + export STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da [e2e-llm-inference-service] + STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da [e2e-llm-inference-service] + COMPONENT_NAME=odh-kserve-llmisvc-controller-ci [e2e-llm-inference-service] ++ jq -r --arg component_name odh-kserve-llmisvc-controller-ci '.[$component_name].image' [e2e-llm-inference-service] + export LLMISVC_CONTROLLER_IMAGE=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:75646d30f65570d55e8a328e38a96abb4134409902fc2e94b8713dc206a6d886 [e2e-llm-inference-service] + LLMISVC_CONTROLLER_IMAGE=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:75646d30f65570d55e8a328e38a96abb4134409902fc2e94b8713dc206a6d886 [e2e-llm-inference-service] + ./test/scripts/openshift-ci/run-e2e-tests.sh 'llminferenceservice and cluster_cpu and not autoscaling and not tracing' 2 llm-d [e2e-llm-inference-service] INFERENCE_POOL_GROUP=inference.networking.k8s.io (detected from OCP 4.21.26) [e2e-llm-inference-service] Installing on cluster [e2e-llm-inference-service] Using namespace: kserve for KServe components [e2e-llm-inference-service] SKLEARN_IMAGE=quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814 [e2e-llm-inference-service] OPT_125M_MODEL_URI=s3://example-models/facebook/opt-125m [e2e-llm-inference-service] ERROR_404_ISVC_IMAGE=quay.io/opendatahub/error-404-isvc:odh-pr-1814 [e2e-llm-inference-service] SUCCESS_200_ISVC_IMAGE=quay.io/opendatahub/success-200-isvc:odh-pr-1814 [e2e-llm-inference-service] [INFO] Installing Kustomize v5.8.1 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed Kustomize v5.8.1 to /workspace/source/bin/kustomize [e2e-llm-inference-service] v5.8.1 [e2e-llm-inference-service] make: Entering directory '/workspace/source' [e2e-llm-inference-service] [INFO] Installing yq v4.52.1 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed yq v4.52.1 to /workspace/source/bin/yq [e2e-llm-inference-service] yq (https://github.com/mikefarah/yq/) version v4.52.1 [e2e-llm-inference-service] make: Leaving directory '/workspace/source' [e2e-llm-inference-service] Installing KServe Python SDK ... [e2e-llm-inference-service] [INFO] Installing uv 0.7.8 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed uv 0.7.8 to /workspace/source/bin/uv [e2e-llm-inference-service] warning: Failed to read project metadata (No `pyproject.toml` found in current directory or any parent directory). Running `uv self version` for compatibility. This fallback will be removed in the future; pass `--preview` to force an error. [e2e-llm-inference-service] uv 0.7.8 [e2e-llm-inference-service] Creating virtual environment... [e2e-llm-inference-service] warning: virtualenv's `--clear` has no effect (uv always clears the virtual environment) [e2e-llm-inference-service] Using CPython 3.9.25 interpreter at: /usr/bin/python3 [e2e-llm-inference-service] Creating virtual environment at: .venv [e2e-llm-inference-service] /workspace/source [e2e-llm-inference-service] Using CPython 3.11.13 interpreter at: /usr/bin/python3.11 [e2e-llm-inference-service] Creating virtual environment at: .venv [e2e-llm-inference-service] Resolved 274 packages in 1ms [e2e-llm-inference-service] Building kserve @ file:///workspace/source/python/kserve [e2e-llm-inference-service] Downloading kubernetes (1.9MiB) [e2e-llm-inference-service] Downloading aiohttp (1.7MiB) [e2e-llm-inference-service] Downloading pandas (12.5MiB) [e2e-llm-inference-service] Downloading cryptography (4.5MiB) [e2e-llm-inference-service] Downloading pydantic-core (2.0MiB) [e2e-llm-inference-service] Downloading numpy (15.7MiB) [e2e-llm-inference-service] Downloading setuptools (1.2MiB) [e2e-llm-inference-service] Downloading uvloop (3.8MiB) [e2e-llm-inference-service] Downloading grpcio-tools (2.5MiB) [e2e-llm-inference-service] Downloading black (1.6MiB) [e2e-llm-inference-service] Downloading botocore (12.9MiB) [e2e-llm-inference-service] Downloading portforward (3.9MiB) [e2e-llm-inference-service] Downloading pyarrow (40.1MiB) [e2e-llm-inference-service] Downloading grpcio (6.4MiB) [e2e-llm-inference-service] Downloading mypy (17.2MiB) [e2e-llm-inference-service] Building timeout-sampler==1.0.3 [e2e-llm-inference-service] Building python-simple-logger==2.0.19 [e2e-llm-inference-service] Downloading aiohttp [e2e-llm-inference-service] Downloading pydantic-core [e2e-llm-inference-service] Downloading black [e2e-llm-inference-service] Downloading grpcio-tools [e2e-llm-inference-service] Downloading setuptools [e2e-llm-inference-service] Built python-simple-logger==2.0.19 [e2e-llm-inference-service] Downloading portforward [e2e-llm-inference-service] Downloading uvloop [e2e-llm-inference-service] Downloading cryptography [e2e-llm-inference-service] Downloading grpcio [e2e-llm-inference-service] Downloading kubernetes [e2e-llm-inference-service] Built timeout-sampler==1.0.3 [e2e-llm-inference-service] Downloading numpy [e2e-llm-inference-service] Built kserve @ file:///workspace/source/python/kserve [e2e-llm-inference-service] Downloading pandas [e2e-llm-inference-service] Downloading botocore [e2e-llm-inference-service] Downloading pyarrow [e2e-llm-inference-service] Downloading mypy [e2e-llm-inference-service] Prepared 101 packages in 2.08s [e2e-llm-inference-service] warning: Failed to hardlink files; falling back to full copy. This may lead to degraded performance. [e2e-llm-inference-service] If the cache and target directories are on different filesystems, hardlinking may not be supported. [e2e-llm-inference-service] If this is intentional, set `export UV_LINK_MODE=copy` or use `--link-mode=copy` to suppress this warning. [e2e-llm-inference-service] Installed 101 packages in 339ms [e2e-llm-inference-service] + aiohappyeyeballs==2.6.1 [e2e-llm-inference-service] + aiohttp==3.14.1 [e2e-llm-inference-service] + aiosignal==1.4.0 [e2e-llm-inference-service] + annotated-doc==0.0.4 [e2e-llm-inference-service] + annotated-types==0.7.0 [e2e-llm-inference-service] + anyio==4.9.0 [e2e-llm-inference-service] + attrs==25.3.0 [e2e-llm-inference-service] + avro==1.12.0 [e2e-llm-inference-service] + black==24.3.0 [e2e-llm-inference-service] + boto3==1.37.35 [e2e-llm-inference-service] + botocore==1.37.35 [e2e-llm-inference-service] + cachetools==5.5.2 [e2e-llm-inference-service] + certifi==2025.1.31 [e2e-llm-inference-service] + cffi==2.0.0 [e2e-llm-inference-service] + charset-normalizer==3.4.1 [e2e-llm-inference-service] + click==8.4.2 [e2e-llm-inference-service] + cloudevents==1.11.0 [e2e-llm-inference-service] + colorama==0.4.6 [e2e-llm-inference-service] + colorlog==6.10.1 [e2e-llm-inference-service] + coverage==7.8.0 [e2e-llm-inference-service] + cryptography==49.0.0 [e2e-llm-inference-service] + deprecation==2.1.0 [e2e-llm-inference-service] + durationpy==0.9 [e2e-llm-inference-service] + execnet==2.1.1 [e2e-llm-inference-service] + fastapi==0.136.3 [e2e-llm-inference-service] + frozenlist==1.5.0 [e2e-llm-inference-service] + google-auth==2.39.0 [e2e-llm-inference-service] + grpc-interceptor==0.15.4 [e2e-llm-inference-service] + grpcio==1.78.1 [e2e-llm-inference-service] + grpcio-testing==1.78.1 [e2e-llm-inference-service] + grpcio-tools==1.78.1 [e2e-llm-inference-service] + h11==0.16.0 [e2e-llm-inference-service] + httpcore==1.0.9 [e2e-llm-inference-service] + httptools==0.6.4 [e2e-llm-inference-service] + httpx==0.27.2 [e2e-llm-inference-service] + httpx-retries==0.4.5 [e2e-llm-inference-service] + idna==3.10 [e2e-llm-inference-service] + iniconfig==2.1.0 [e2e-llm-inference-service] + jinja2==3.1.6 [e2e-llm-inference-service] + jmespath==1.0.1 [e2e-llm-inference-service] + kserve==0.20.0rc0 (from file:///workspace/source/python/kserve) [e2e-llm-inference-service] + kubernetes==32.0.1 [e2e-llm-inference-service] + markupsafe==3.0.2 [e2e-llm-inference-service] + multidict==6.4.3 [e2e-llm-inference-service] + mypy==0.991 [e2e-llm-inference-service] + mypy-extensions==1.0.0 [e2e-llm-inference-service] + numpy==2.2.4 [e2e-llm-inference-service] + oauthlib==3.2.2 [e2e-llm-inference-service] + orjson==3.11.9 [e2e-llm-inference-service] + packaging==24.2 [e2e-llm-inference-service] + pandas==2.2.3 [e2e-llm-inference-service] + pathspec==0.12.1 [e2e-llm-inference-service] + platformdirs==4.3.7 [e2e-llm-inference-service] + pluggy==1.5.0 [e2e-llm-inference-service] + portforward==0.7.1 [e2e-llm-inference-service] + prometheus-client==0.21.1 [e2e-llm-inference-service] + propcache==0.3.1 [e2e-llm-inference-service] + protobuf==6.33.5 [e2e-llm-inference-service] + psutil==5.9.8 [e2e-llm-inference-service] + pyarrow==19.0.1 [e2e-llm-inference-service] + pyasn1==0.6.3 [e2e-llm-inference-service] + pyasn1-modules==0.4.2 [e2e-llm-inference-service] + pycparser==2.22 [e2e-llm-inference-service] + pydantic==2.12.4 [e2e-llm-inference-service] + pydantic-core==2.41.5 [e2e-llm-inference-service] + pyjwt==2.13.0 [e2e-llm-inference-service] + pytest==7.4.4 [e2e-llm-inference-service] + pytest-asyncio==0.23.8 [e2e-llm-inference-service] + pytest-cov==5.0.0 [e2e-llm-inference-service] + pytest-httpx==0.30.0 [e2e-llm-inference-service] + pytest-json-report==1.5.0 [e2e-llm-inference-service] + pytest-metadata==3.1.1 [e2e-llm-inference-service] + pytest-xdist==3.6.1 [e2e-llm-inference-service] + python-dateutil==2.9.0.post0 [e2e-llm-inference-service] + python-dotenv==1.1.0 [e2e-llm-inference-service] + python-multipart==0.0.32 [e2e-llm-inference-service] + python-simple-logger==2.0.19 [e2e-llm-inference-service] + pytz==2025.2 [e2e-llm-inference-service] + pyyaml==6.0.2 [e2e-llm-inference-service] + requests==2.32.3 [e2e-llm-inference-service] + requests-oauthlib==2.0.0 [e2e-llm-inference-service] + rsa==4.9.1 [e2e-llm-inference-service] + s3transfer==0.11.4 [e2e-llm-inference-service] + setuptools==78.1.0 [e2e-llm-inference-service] + six==1.17.0 [e2e-llm-inference-service] + sniffio==1.3.1 [e2e-llm-inference-service] + starlette==1.2.1 [e2e-llm-inference-service] + tabulate==0.9.0 [e2e-llm-inference-service] + timeout-sampler==1.0.3 [e2e-llm-inference-service] + timing-asgi==0.3.1 [e2e-llm-inference-service] + tomlkit==0.13.2 [e2e-llm-inference-service] + typing-extensions==4.15.0 [e2e-llm-inference-service] + typing-inspection==0.4.2 [e2e-llm-inference-service] + tzdata==2025.2 [e2e-llm-inference-service] + urllib3==2.7.0 [e2e-llm-inference-service] + uvicorn==0.34.1 [e2e-llm-inference-service] + uvloop==0.21.0 [e2e-llm-inference-service] + watchfiles==1.0.5 [e2e-llm-inference-service] + websocket-client==1.8.0 [e2e-llm-inference-service] + websockets==15.0.1 [e2e-llm-inference-service] + yarl==1.20.0 [e2e-llm-inference-service] Audited 1 package in 48ms [e2e-llm-inference-service] /workspace/source [e2e-llm-inference-service] [INFO] Installing Kustomize v5.8.1 for linux/amd64... [e2e-llm-inference-service] [INFO] Kustomize v5.8.1 is already installed in /workspace/source/bin (>= v5.8.1) [e2e-llm-inference-service] make: Entering directory '/workspace/source' [e2e-llm-inference-service] make: Leaving directory '/workspace/source' [e2e-llm-inference-service] Now using project "kserve" on server "https://afd3bdff287004064be9628906775d0c-001fd88cd0fa6e57.elb.us-east-1.amazonaws.com:6443". [e2e-llm-inference-service] [e2e-llm-inference-service] You can add applications to this project with the 'new-app' command. For example, try: [e2e-llm-inference-service] [e2e-llm-inference-service] oc new-app rails-postgresql-example [e2e-llm-inference-service] [e2e-llm-inference-service] to build a new example application in Ruby. Or use kubectl to deploy a simple Kubernetes application: [e2e-llm-inference-service] [e2e-llm-inference-service] kubectl create deployment hello-node --image=registry.k8s.io/e2e-test-images/agnhost:2.43 -- /agnhost serve-hostname [e2e-llm-inference-service] [e2e-llm-inference-service] [INFO] Installing Kustomize v5.8.1 for linux/amd64... [e2e-llm-inference-service] [INFO] Kustomize v5.8.1 is already installed in /workspace/source/bin (>= v5.8.1) [e2e-llm-inference-service] make: Entering directory '/workspace/source' [e2e-llm-inference-service] make: Leaving directory '/workspace/source' [e2e-llm-inference-service] Creating namespace openshift-keda... [e2e-llm-inference-service] namespace/openshift-keda created [e2e-llm-inference-service] Namespace openshift-keda created/ensured. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Creating OperatorGroup openshift-keda... [e2e-llm-inference-service] operatorgroup.operators.coreos.com/openshift-keda created [e2e-llm-inference-service] OperatorGroup openshift-keda created/ensured. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Creating Subscription for openshift-custom-metrics-autoscaler-operator... [e2e-llm-inference-service] subscription.operators.coreos.com/openshift-custom-metrics-autoscaler-operator created [e2e-llm-inference-service] Subscription openshift-custom-metrics-autoscaler-operator created/ensured. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Waiting for openshift-custom-metrics-autoscaler-operator CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (0/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (5/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (10/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (15/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (20/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-2 found, but not yet Succeeded (Phase: Installing). Waiting... (25/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-2 found, but not yet Succeeded (Phase: Installing). Waiting... (30/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-2 found, but not yet Succeeded (Phase: Installing). Waiting... (35/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-2 found, but not yet Succeeded (Phase: Installing). Waiting... (40/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-2 found, but not yet Succeeded (Phase: Installing). Waiting... (45/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-2 is ready (Phase: Succeeded). [e2e-llm-inference-service] --- [e2e-llm-inference-service] Applying KedaController custom resource... [e2e-llm-inference-service] Warning: resource kedacontrollers/keda is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by oc apply. oc apply should only be used on resources created declaratively by either oc create --save-config or oc apply. The missing annotation will be patched automatically. [e2e-llm-inference-service] kedacontroller.keda.sh/keda configured [e2e-llm-inference-service] KedaController custom resource applied. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Allowing time for KEDA components to be provisioned by the operator ... [e2e-llm-inference-service] Waiting for KEDA Operator pod (selector: "app=keda-operator") to be ready in namespace openshift-keda... [e2e-llm-inference-service] Waiting for pod -l "app=keda-operator" in namespace "openshift-keda" to be created... [e2e-llm-inference-service] Pod -l "app=keda-operator" in namespace "openshift-keda" found. [e2e-llm-inference-service] Current pods for -l "app=keda-operator" in namespace "openshift-keda": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] keda-operator-77f648d694-p4kwp 1/1 Running 0 40s [e2e-llm-inference-service] Waiting up to 120s for pod(s) -l "app=keda-operator" in namespace "openshift-keda" to become ready... [e2e-llm-inference-service] pod/keda-operator-77f648d694-p4kwp condition met [e2e-llm-inference-service] Pod(s) -l "app=keda-operator" in namespace "openshift-keda" are ready. [e2e-llm-inference-service] KEDA Operator pod is ready. [e2e-llm-inference-service] Waiting for KEDA Metrics API Server pod (selector: "app=keda-metrics-apiserver") to be ready in namespace openshift-keda... [e2e-llm-inference-service] Waiting for pod -l "app=keda-metrics-apiserver" in namespace "openshift-keda" to be created... [e2e-llm-inference-service] Pod -l "app=keda-metrics-apiserver" in namespace "openshift-keda" found. [e2e-llm-inference-service] Current pods for -l "app=keda-metrics-apiserver" in namespace "openshift-keda": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] keda-metrics-apiserver-55557df8c-lq4zt 1/1 Running 0 46s [e2e-llm-inference-service] Waiting up to 120s for pod(s) -l "app=keda-metrics-apiserver" in namespace "openshift-keda" to become ready... [e2e-llm-inference-service] pod/keda-metrics-apiserver-55557df8c-lq4zt condition met [e2e-llm-inference-service] Pod(s) -l "app=keda-metrics-apiserver" in namespace "openshift-keda" are ready. [e2e-llm-inference-service] KEDA Metrics API Server pod is ready. [e2e-llm-inference-service] Waiting for KEDA Webhook pod (selector: "app=keda-admission-webhooks") to be ready in namespace openshift-keda... [e2e-llm-inference-service] Waiting for pod -l "app=keda-admission-webhooks" in namespace "openshift-keda" to be created... [e2e-llm-inference-service] Pod -l "app=keda-admission-webhooks" in namespace "openshift-keda" found. [e2e-llm-inference-service] Current pods for -l "app=keda-admission-webhooks" in namespace "openshift-keda": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] keda-admission-56f9798bb4-l9spq 1/1 Running 0 51s [e2e-llm-inference-service] Waiting up to 120s for pod(s) -l "app=keda-admission-webhooks" in namespace "openshift-keda" to become ready... [e2e-llm-inference-service] pod/keda-admission-56f9798bb4-l9spq condition met [e2e-llm-inference-service] Pod(s) -l "app=keda-admission-webhooks" in namespace "openshift-keda" are ready. [e2e-llm-inference-service] KEDA Webhook pod is ready. [e2e-llm-inference-service] --- [e2e-llm-inference-service] ✅ KEDA deployment script finished successfully. [e2e-llm-inference-service] KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:e84ef678972994166b9c11fc0192a03ca1e4fecf65c513bcc81ed8eba01106de [e2e-llm-inference-service] LLMISVC_CONTROLLER_IMAGE=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:75646d30f65570d55e8a328e38a96abb4134409902fc2e94b8713dc206a6d886 [e2e-llm-inference-service] KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:efaf0bd4be5e59e0fd5df072416f565ae616956e4a23f814ea31d63b5efa6d4b [e2e-llm-inference-service] KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:24a06dcfbbe7f0d25115c5a79181136502c6b1ca34878eb57638d7eb57f020b2 [e2e-llm-inference-service] STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da [e2e-llm-inference-service] Installing KServe via kustomize... [e2e-llm-inference-service] === Final params.env [e2e-llm-inference-service] kserve-controller=quay.io/opendatahub/kserve-controller@sha256:e84ef678972994166b9c11fc0192a03ca1e4fecf65c513bcc81ed8eba01106de [e2e-llm-inference-service] llmisvc-controller=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:75646d30f65570d55e8a328e38a96abb4134409902fc2e94b8713dc206a6d886 [e2e-llm-inference-service] kserve-agent=quay.io/opendatahub/kserve-agent@sha256:efaf0bd4be5e59e0fd5df072416f565ae616956e4a23f814ea31d63b5efa6d4b [e2e-llm-inference-service] kserve-router=quay.io/opendatahub/kserve-router@sha256:24a06dcfbbe7f0d25115c5a79181136502c6b1ca34878eb57638d7eb57f020b2 [e2e-llm-inference-service] kserve-storage-initializer=quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da [e2e-llm-inference-service] kserve-llm-d=registry.redhat.io/rhaiis/vllm-cuda-rhel9@sha256:fc68d623d1bfc36c8cb2fe4a71f19c8578cfb420ce8ce07b20a02c1ee0be0cf3 [e2e-llm-inference-service] kserve-llm-d-inference-scheduler=quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] kserve-llm-d-routing-sidecar=quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.9.0 [e2e-llm-inference-service] kserve-llm-d-uds-tokenizer=quay.io/opendatahub/llm-d-kv-cache:v0.8.0 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda=registry.redhat.io/rhaiis/vllm-cuda-rhel9@sha256:fc68d623d1bfc36c8cb2fe4a71f19c8578cfb420ce8ce07b20a02c1ee0be0cf3 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda-fast-1=registry.redhat.io/rhaiis/vllm-cuda-rhel9@sha256:fc68d623d1bfc36c8cb2fe4a71f19c8578cfb420ce8ce07b20a02c1ee0be0cf3 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda-fast-2=registry.redhat.io/rhaiis/vllm-cuda-rhel9@sha256:fc68d623d1bfc36c8cb2fe4a71f19c8578cfb420ce8ce07b20a02c1ee0be0cf3 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda-fast-1-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda-fast-2-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-amd-rocm=registry.redhat.io/rhaiis/vllm-rocm-rhel9@sha256:d9a48add238cc095fa43eeee17c8c4d104de60c4dc623e0bc7f8c4b53b2b2e97 [e2e-llm-inference-service] kserve-llm-d-amd-rocm-fast-1=registry.redhat.io/rhaiis/vllm-rocm-rhel9@sha256:d9a48add238cc095fa43eeee17c8c4d104de60c4dc623e0bc7f8c4b53b2b2e97 [e2e-llm-inference-service] kserve-llm-d-amd-rocm-fast-2=registry.redhat.io/rhaiis/vllm-rocm-rhel9@sha256:d9a48add238cc095fa43eeee17c8c4d104de60c4dc623e0bc7f8c4b53b2b2e97 [e2e-llm-inference-service] kserve-llm-d-amd-rocm-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-amd-rocm-fast-1-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-amd-rocm-fast-2-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi=registry.redhat.io/rhaii-early-access/vllm-gaudi-rhel9:3.4.0-ea.2 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi-fast-1=registry.redhat.io/rhaii-early-access/vllm-gaudi-rhel9:3.4.0-ea.2 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi-fast-2=registry.redhat.io/rhaii-early-access/vllm-gaudi-rhel9:3.4.0-ea.2 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi-upstream-version=0.16.0 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi-fast-1-upstream-version=0.16.0 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi-fast-2-upstream-version=0.16.0 [e2e-llm-inference-service] kserve-llm-d-ibm-spyre=registry.redhat.io/rhaiis/vllm-spyre-rhel9@sha256:80ae3e435a5be2c1f117f36599103ab05357917dd6e37f0df6613cb3ac2c13ea [e2e-llm-inference-service] kserve-llm-d-ibm-spyre-fast-1=registry.redhat.io/rhaiis/vllm-spyre-rhel9@sha256:80ae3e435a5be2c1f117f36599103ab05357917dd6e37f0df6613cb3ac2c13ea [e2e-llm-inference-service] kserve-llm-d-ibm-spyre-fast-2=registry.redhat.io/rhaiis/vllm-spyre-rhel9@sha256:80ae3e435a5be2c1f117f36599103ab05357917dd6e37f0df6613cb3ac2c13ea [e2e-llm-inference-service] kserve-llm-d-ibm-spyre-upstream-version=0.10.2 [e2e-llm-inference-service] kserve-llm-d-ibm-spyre-fast-1-upstream-version=0.10.2 [e2e-llm-inference-service] kserve-llm-d-ibm-spyre-fast-2-upstream-version=0.10.2 [e2e-llm-inference-service] kserve-llm-d-latency-predictor-prediction=quay.io/opendatahub/odh-latency-predictor-prediction:odh-stable [e2e-llm-inference-service] kserve-llm-d-latency-predictor-training=quay.io/opendatahub/odh-latency-predictor-training:odh-stable [e2e-llm-inference-service] # TODO update when our changes are introduced in the official image [e2e-llm-inference-service] kube-rbac-proxy=quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3 [e2e-llm-inference-service] kserve-localmodel-controller=quay.io/opendatahub/odh-kserve-localmodel-controller:odh-stable [e2e-llm-inference-service] kserve-localmodelnode-agent=quay.io/opendatahub/odh-kserve-localmodelnode-agent:odh-stable [e2e-llm-inference-service] ovms-versioning-ubi-micro=registry.redhat.io/ubi9/ubi-micro@sha256:38e934147827349f2b8b11ac9c38d7be23cb8e29f128b1c46e5c7ae54a2d23cd [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/dscinitializations.dscinitialization.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencemodelrewrites.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceobjectives.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-llm-inference-service] Waiting for CRDs to be established... [e2e-llm-inference-service] Waiting for CRD inferenceservices.serving.kserve.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD inferenceservices.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io condition met [e2e-llm-inference-service] Waiting for CRD llminferenceserviceconfigs.serving.kserve.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD llminferenceserviceconfigs.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io condition met [e2e-llm-inference-service] Waiting for CRD clusterstoragecontainers.serving.kserve.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD clusterstoragecontainers.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io condition met [e2e-llm-inference-service] Waiting for CRD datascienceclusters.datasciencecluster.opendatahub.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD datascienceclusters.datasciencecluster.opendatahub.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io condition met [e2e-llm-inference-service] Applying all resources... [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/dscinitializations.dscinitialization.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencemodelrewrites.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceobjectives.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-llm-inference-service] serviceaccount/kserve-controller-manager serverside-applied [e2e-llm-inference-service] serviceaccount/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/kserve-leader-election-role serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/kserve-llmisvcconfig-read-access serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/llmisvc-leader-election-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-admin serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-edit serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-inferenceservice-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-manager-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-manager-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-metrics-reader-cluster-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-models-admin serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-models-edit serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-models-view serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-proxy-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-tls-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-view serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/openshift-ai-inferenceservice-image-volume-scc serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/kserve-leader-election-rolebinding serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/kserve-llmisvcconfig-read-access serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/llmisvc-leader-election-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-inferenceservice-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-llmisvc-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-manager-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-proxy-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-tls-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/llmisvc-manager-rolebinding serverside-applied [e2e-llm-inference-service] configmap/inferenceservice-config serverside-applied [e2e-llm-inference-service] configmap/kserve-parameters serverside-applied [e2e-llm-inference-service] secret/kserve-webhook-server-secret serverside-applied [e2e-llm-inference-service] secret/mlpipeline-s3-artifact serverside-applied [e2e-llm-inference-service] service/kserve-controller-manager-metrics-service serverside-applied [e2e-llm-inference-service] service/kserve-controller-manager-service serverside-applied [e2e-llm-inference-service] service/kserve-webhook-server-service serverside-applied [e2e-llm-inference-service] service/llmisvc-controller-manager-service serverside-applied [e2e-llm-inference-service] service/llmisvc-webhook-server-service serverside-applied [e2e-llm-inference-service] service/s3-service serverside-applied [e2e-llm-inference-service] deployment.apps/kserve-controller-manager serverside-applied [e2e-llm-inference-service] deployment.apps/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] deployment.apps/seaweedfs serverside-applied [e2e-llm-inference-service] networkpolicy.networking.k8s.io/kserve-controller-manager serverside-applied [e2e-llm-inference-service] securitycontextconstraints.security.openshift.io/openshift-ai-inferenceservice-image-volume-scc serverside-applied [e2e-llm-inference-service] securitycontextconstraints.security.openshift.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-llm-inference-service] clusterstoragecontainer.serving.kserve.io/default serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-router-route serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler-latency-predictor serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-tokenizer serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-tracing serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-worker-data-parallel serverside-applied [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/inferencegraph.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceserviceconfig.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/servingruntime.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/trainedmodel.serving.kserve.io serverside-applied [e2e-llm-inference-service] Waiting for llmisvc-controller-manager to be ready... [e2e-llm-inference-service] Waiting for pod -l "control-plane=llmisvc-controller-manager" in namespace "kserve" to be created... [e2e-llm-inference-service] Pod -l "control-plane=llmisvc-controller-manager" in namespace "kserve" found. [e2e-llm-inference-service] Current pods for -l "control-plane=llmisvc-controller-manager" in namespace "kserve": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] llmisvc-controller-manager-84d9876847-shpsd 0/1 Running 0 6s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "control-plane=llmisvc-controller-manager" in namespace "kserve" to become ready... [e2e-llm-inference-service] pod/llmisvc-controller-manager-84d9876847-shpsd condition met [e2e-llm-inference-service] Pod(s) -l "control-plane=llmisvc-controller-manager" in namespace "kserve" are ready. [e2e-llm-inference-service] Re-Applying all resources... [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/dscinitializations.dscinitialization.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencemodelrewrites.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceobjectives.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-llm-inference-service] serviceaccount/kserve-controller-manager serverside-applied [e2e-llm-inference-service] serviceaccount/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/kserve-leader-election-role serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/kserve-llmisvcconfig-read-access serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/llmisvc-leader-election-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-admin serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-edit serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-inferenceservice-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-manager-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-manager-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-metrics-reader-cluster-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-models-admin serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-models-edit serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-models-view serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-proxy-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-tls-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-view serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/openshift-ai-inferenceservice-image-volume-scc serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/kserve-leader-election-rolebinding serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/kserve-llmisvcconfig-read-access serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/llmisvc-leader-election-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-inferenceservice-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-llmisvc-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-manager-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-proxy-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-tls-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/llmisvc-manager-rolebinding serverside-applied [e2e-llm-inference-service] configmap/inferenceservice-config serverside-applied [e2e-llm-inference-service] configmap/kserve-parameters serverside-applied [e2e-llm-inference-service] secret/kserve-webhook-server-secret serverside-applied [e2e-llm-inference-service] secret/mlpipeline-s3-artifact serverside-applied [e2e-llm-inference-service] service/kserve-controller-manager-metrics-service serverside-applied [e2e-llm-inference-service] service/kserve-controller-manager-service serverside-applied [e2e-llm-inference-service] service/kserve-webhook-server-service serverside-applied [e2e-llm-inference-service] service/llmisvc-controller-manager-service serverside-applied [e2e-llm-inference-service] service/llmisvc-webhook-server-service serverside-applied [e2e-llm-inference-service] service/s3-service serverside-applied [e2e-llm-inference-service] deployment.apps/kserve-controller-manager serverside-applied [e2e-llm-inference-service] deployment.apps/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] deployment.apps/seaweedfs serverside-applied [e2e-llm-inference-service] networkpolicy.networking.k8s.io/kserve-controller-manager serverside-applied [e2e-llm-inference-service] securitycontextconstraints.security.openshift.io/openshift-ai-inferenceservice-image-volume-scc serverside-applied [e2e-llm-inference-service] securitycontextconstraints.security.openshift.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-llm-inference-service] clusterstoragecontainer.serving.kserve.io/default serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-router-route serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler-latency-predictor serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-tokenizer serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-tracing serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-worker-data-parallel serverside-applied [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/inferencegraph.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceserviceconfig.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/servingruntime.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/trainedmodel.serving.kserve.io serverside-applied [e2e-llm-inference-service] Applying DSC/DSCI resources... [e2e-llm-inference-service] dscinitialization.dscinitialization.opendatahub.io/test-dsci created [e2e-llm-inference-service] datasciencecluster.datasciencecluster.opendatahub.io/test-dsc created [e2e-llm-inference-service] KServe manual installation complete [e2e-llm-inference-service] 🔧 Configuration: [e2e-llm-inference-service] KServe deployment: ❌ disabled [e2e-llm-inference-service] Kuadrant deployment: ✅ enabled [e2e-llm-inference-service] [e2e-llm-inference-service] Checking OpenShift server version...(4.21.26) [e2e-llm-inference-service] 🎯 Server version (4.21.26) is 4.19.9 or higher - continue with the script [e2e-llm-inference-service] ⏳ Installing cert-manager [e2e-llm-inference-service] namespace/cert-manager-operator created [e2e-llm-inference-service] operatorgroup.operators.coreos.com/openshift-cert-manager-operator created [e2e-llm-inference-service] subscription.operators.coreos.com/openshift-cert-manager-operator created [e2e-llm-inference-service] Waiting for openshift-cert-manager-operator CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (0/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (5/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (10/300) [e2e-llm-inference-service] CSV cert-manager-operator.v1.20.0 found, but not yet Succeeded (Phase: Installing). Waiting... (15/300) [e2e-llm-inference-service] CSV cert-manager-operator.v1.20.0 is ready (Phase: Succeeded). [e2e-llm-inference-service] Waiting for CRD certificates.cert-manager.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD certificates.cert-manager.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io condition met [e2e-llm-inference-service] ✅ Cert-manager installed [e2e-llm-inference-service] ⏳ Installing openshift-lws-operator [e2e-llm-inference-service] namespace/openshift-lws-operator created [e2e-llm-inference-service] operatorgroup.operators.coreos.com/leader-worker-set created [e2e-llm-inference-service] subscription.operators.coreos.com/leader-worker-set created [e2e-llm-inference-service] Waiting for leader-worker-set CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (0/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (5/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (10/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (15/300) [e2e-llm-inference-service] CSV leader-worker-set.v1.0.0 is ready (Phase: Succeeded). [e2e-llm-inference-service] Waiting for CRD leaderworkersetoperators.operator.openshift.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD leaderworkersetoperators.operator.openshift.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/leaderworkersetoperators.operator.openshift.io condition met [e2e-llm-inference-service] leaderworkersetoperator.operator.openshift.io/cluster created [e2e-llm-inference-service] ⏳ waiting for openshift-lws-operator to be ready.… [e2e-llm-inference-service] Waiting for pod -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" to be created... [e2e-llm-inference-service] Pod -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" found. [e2e-llm-inference-service] Current pods for -l "name=openshift-lws-operator" in namespace "openshift-lws-operator": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] openshift-lws-operator-fd8ccff4c-9978k 1/1 Running 0 12s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" to become ready... [e2e-llm-inference-service] pod/openshift-lws-operator-fd8ccff4c-9978k condition met [e2e-llm-inference-service] Pod(s) -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" are ready. [e2e-llm-inference-service] ✅ openshift-lws-operator installed [e2e-llm-inference-service] gatewayclass.gateway.networking.k8s.io/openshift-default created [e2e-llm-inference-service] Waiting for pod -l "app=istiod" in namespace "openshift-ingress" to be created... [e2e-llm-inference-service] Pod -l "app=istiod" in namespace "openshift-ingress" found. [e2e-llm-inference-service] Current pods for -l "app=istiod" in namespace "openshift-ingress": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] istiod-openshift-gateway-5677ccb49b-bv8fm 1/1 Running 0 6s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "app=istiod" in namespace "openshift-ingress" to become ready... [e2e-llm-inference-service] pod/istiod-openshift-gateway-5677ccb49b-bv8fm condition met [e2e-llm-inference-service] Pod(s) -l "app=istiod" in namespace "openshift-ingress" are ready. [e2e-llm-inference-service] ⏳ Creating a Gateway [e2e-llm-inference-service] Error from server (AlreadyExists): namespaces "openshift-ingress" already exists [e2e-llm-inference-service] ⏳ Creating gateway memory ConfigMap for parametersRef (2Gi) [e2e-llm-inference-service] configmap/gateway-proxy-config created [e2e-llm-inference-service] gateway.gateway.networking.k8s.io/openshift-ai-inference created [e2e-llm-inference-service] Waiting for pod -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" to be created... [e2e-llm-inference-service] Pod -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" found. [e2e-llm-inference-service] Current pods for -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l 1/1 Running 0 5s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" to become ready... [e2e-llm-inference-service] pod/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l condition met [e2e-llm-inference-service] Pod(s) -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" are ready. [e2e-llm-inference-service] ⏳ Installing RHCL(Kuadrant) operator [e2e-llm-inference-service] namespace/kuadrant-system created [e2e-llm-inference-service] subscription.operators.coreos.com/rhcl-operator created [e2e-llm-inference-service] operatorgroup.operators.coreos.com/kuadrant created [e2e-llm-inference-service] Waiting for rhcl-operator CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (0/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (5/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (10/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (15/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.2 found, but not yet Succeeded (Phase: Pending). Waiting... (20/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.2 found, but not yet Succeeded (Phase: Installing). Waiting... (25/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.2 found, but not yet Succeeded (Phase: Installing). Waiting... (30/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.2 found, but not yet Succeeded (Phase: Installing). Waiting... (35/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.2 is ready (Phase: Succeeded). [e2e-llm-inference-service] Waiting for CRD kuadrants.kuadrant.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD kuadrants.kuadrant.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/kuadrants.kuadrant.io condition met [e2e-llm-inference-service] Waiting for apiserver discovery /apis/kuadrant.io/v1beta1 to list kuadrants (timeout: 120s)… [e2e-llm-inference-service] Discovery for kuadrant.io/v1beta1 includes kuadrants. [e2e-llm-inference-service] ⏳ sleeping 30s after discovery (RESTMapper can trail discovery)… [e2e-llm-inference-service] kuadrant.kuadrant.io/kuadrant created [e2e-llm-inference-service] ⏳ waiting for Kuadrant Ready (attempt 1/2, timeout 5m)… [e2e-llm-inference-service] kuadrant.kuadrant.io/kuadrant condition met [e2e-llm-inference-service] Waiting for pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" to be created... [e2e-llm-inference-service] Pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" found. [e2e-llm-inference-service] Current pods for -l "control-plane=authorino-operator" in namespace "kuadrant-system": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] authorino-operator-56fb6f6856-2kw2j 1/1 Running 0 77s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" to become ready... [e2e-llm-inference-service] pod/authorino-operator-56fb6f6856-2kw2j condition met [e2e-llm-inference-service] Pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" are ready. [e2e-llm-inference-service] ⏳ waiting for authorino service to be created... [e2e-llm-inference-service] service/authorino-authorino-authorization condition met [e2e-llm-inference-service] service/authorino-authorino-authorization annotated [e2e-llm-inference-service] Warning: resource authorinos/authorino is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by oc apply. oc apply should only be used on resources created declaratively by either oc create --save-config or oc apply. The missing annotation will be patched automatically. [e2e-llm-inference-service] authorino.operator.authorino.kuadrant.io/authorino configured [e2e-llm-inference-service] Waiting for pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" to be created... [e2e-llm-inference-service] Pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" found. [e2e-llm-inference-service] Current pods for -l "control-plane=authorino-operator" in namespace "kuadrant-system": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] authorino-operator-56fb6f6856-2kw2j 1/1 Running 0 87s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" to become ready... [e2e-llm-inference-service] pod/authorino-operator-56fb6f6856-2kw2j condition met [e2e-llm-inference-service] Pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" are ready. [e2e-llm-inference-service] ✅ kuadrant(authorino) installed [e2e-llm-inference-service] Setting up Jaeger (Helm) for LLMISVC tracing e2e... [e2e-llm-inference-service] ⏳ Installing Jaeger All-in-One (Helm) into namespace observability [e2e-llm-inference-service] [INFO] Installing Helm v3.16.3 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed Helm v3.16.3 to /workspace/source/bin/helm [e2e-llm-inference-service] WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /credentials/cluster-g6fqs-kubeconfig [e2e-llm-inference-service] WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /credentials/cluster-g6fqs-kubeconfig [e2e-llm-inference-service] version.BuildInfo{Version:"v3.16.3", GitCommit:"cfd07493f46efc9debd9cc1b02a0961186df7fdf", GitTreeState:"clean", GoVersion:"go1.22.7"} [e2e-llm-inference-service] [INFO] Adding Jaeger Helm repository... [e2e-llm-inference-service] WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /credentials/cluster-g6fqs-kubeconfig [e2e-llm-inference-service] WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /credentials/cluster-g6fqs-kubeconfig [e2e-llm-inference-service] "jaegertracing" has been added to your repositories [e2e-llm-inference-service] [INFO] Installing Jaeger All-in-One 4.7.0... [e2e-llm-inference-service] WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /credentials/cluster-g6fqs-kubeconfig [e2e-llm-inference-service] WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /credentials/cluster-g6fqs-kubeconfig [e2e-llm-inference-service] NAME: jaeger [e2e-llm-inference-service] LAST DEPLOYED: Tue Jul 28 17:26:30 2026 [e2e-llm-inference-service] NAMESPACE: observability [e2e-llm-inference-service] STATUS: deployed [e2e-llm-inference-service] REVISION: 1 [e2e-llm-inference-service] TEST SUITE: None [e2e-llm-inference-service] NOTES: [e2e-llm-inference-service] ################################################################### [e2e-llm-inference-service] ### ⚠️ EXPERIMENTAL - NO STABILITY GUARANTEES ### [e2e-llm-inference-service] ### ### [e2e-llm-inference-service] ### This chart is under active development. ### [e2e-llm-inference-service] ### Breaking changes may occur in minor versions. ### [e2e-llm-inference-service] ### ### [e2e-llm-inference-service] ### See README.md for configuration details. ### [e2e-llm-inference-service] ################################################################### [e2e-llm-inference-service] [e2e-llm-inference-service] 🚀 Congratulations on successfully installing Jaeger v2.17.0 (Chart v4.7.0)! [e2e-llm-inference-service] [e2e-llm-inference-service] To access the query UI: [e2e-llm-inference-service] export POD_NAME=$(kubectl get pods --namespace observability -l "app.kubernetes.io/instance=jaeger,app.kubernetes.io/component=all-in-one" -o jsonpath="{.items[0].metadata.name}") [e2e-llm-inference-service] kubectl port-forward --namespace observability $POD_NAME 16686:16686 --address 0.0.0.0 & [e2e-llm-inference-service] Visit http://127.0.0.1:16686/ [e2e-llm-inference-service] [SUCCESS] Successfully installed Jaeger All-in-One via Helm [e2e-llm-inference-service] [INFO] Waiting for pods with label 'app.kubernetes.io/name=jaeger' in namespace 'observability' to be created... [e2e-llm-inference-service] [INFO] Found 1 pod(s) with label 'app.kubernetes.io/name=jaeger' [e2e-llm-inference-service] [INFO] Waiting for pods with label 'app.kubernetes.io/name=jaeger' in namespace 'observability' to be ready... [e2e-llm-inference-service] pod/jaeger-6bbd7d4c9d-n8f2l condition met [e2e-llm-inference-service] [SUCCESS] Pods with label 'app.kubernetes.io/name=jaeger' in namespace 'observability' are ready! [e2e-llm-inference-service] [SUCCESS] Jaeger is ready! [e2e-llm-inference-service] ⏳ Verifying Jaeger Service ports (OTLP 4317, Query 16686)... [e2e-llm-inference-service] ✅ Jaeger (Helm) ready — OTLP http://jaeger.observability.svc.cluster.local:4317 [e2e-llm-inference-service] Patching ingress domain... [e2e-llm-inference-service] configmap/inferenceservice-config patched [e2e-llm-inference-service] pod "kserve-controller-manager-85d8b8ddfc-dv2cm" deleted [e2e-llm-inference-service] Waiting for kserve-controller-manager to be ready... [e2e-llm-inference-service] pod/kserve-controller-manager-85d8b8ddfc-svxs4 condition met [e2e-llm-inference-service] Installing ODH Model Controller manually... [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/accounts.nim.opendatahub.io created [e2e-llm-inference-service] serviceaccount/model-serving-api created [e2e-llm-inference-service] serviceaccount/odh-model-controller created [e2e-llm-inference-service] role.rbac.authorization.k8s.io/leader-election-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/account-editor-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/account-viewer-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-prometheus-k8s created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/metrics-reader created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/model-serving-api created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/odh-model-controller-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/proxy-role created [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/leader-election-rolebinding created [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/model-serving-api created [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/odh-model-controller-rolebinding-opendatahub created [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/proxy-rolebinding created [e2e-llm-inference-service] configmap/odh-model-controller-parameters created [e2e-llm-inference-service] service/model-serving-api created [e2e-llm-inference-service] service/odh-model-controller-metrics-service created [e2e-llm-inference-service] service/odh-model-controller-webhook-service created [e2e-llm-inference-service] deployment.apps/model-serving-api created [e2e-llm-inference-service] deployment.apps/odh-model-controller created [e2e-llm-inference-service] servicemonitor.monitoring.coreos.com/model-serving-api-metrics created [e2e-llm-inference-service] servicemonitor.monitoring.coreos.com/odh-model-controller-metrics-monitor created [e2e-llm-inference-service] template.template.openshift.io/autogluon-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/guardrails-detector-huggingface-serving-template created [e2e-llm-inference-service] template.template.openshift.io/kserve-ovms created [e2e-llm-inference-service] template.template.openshift.io/mlserver-cuda-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/mlserver-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-x86-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-x86-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-x86-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cuda-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cuda-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cuda-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-gaudi-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-gaudi-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-gaudi-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-multinode-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-multinode-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-multinode-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-rocm-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-rocm-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-rocm-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-ppc64le-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-ppc64le-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-ppc64le-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-s390x-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-s390x-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-s390x-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-x86-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-x86-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-x86-runtime-template-fast-2 created [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/mutating.odh-model-controller.opendatahub.io created [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/validating.odh-model-controller.opendatahub.io created [e2e-llm-inference-service] Waiting for deployment "odh-model-controller" rollout to finish: 0 of 1 updated replicas are available... [e2e-llm-inference-service] deployment "odh-model-controller" successfully rolled out [e2e-llm-inference-service] networkpolicy.networking.k8s.io/allow-all created [e2e-llm-inference-service] KServe setup complete (namespace: kserve) [e2e-llm-inference-service] Add testing models to SeaweedFS S3 storage ... [e2e-llm-inference-service] Waiting for SeaweedFS deployment to be ready... [e2e-llm-inference-service] deployment "seaweedfs" successfully rolled out [e2e-llm-inference-service] S3 init job not completed, re-creating... [e2e-llm-inference-service] job.batch/s3-init replaced [e2e-llm-inference-service] Waiting for S3 init job to complete... [e2e-llm-inference-service] job.batch/s3-init condition met [e2e-llm-inference-service] Prepare CI namespace and install ServingRuntimes [e2e-llm-inference-service] Setting up CI namespace: kserve-ci-e2e-test [e2e-llm-inference-service] Tearing down CI namespace: kserve-ci-e2e-test [e2e-llm-inference-service] Namespace kserve-ci-e2e-test does not exist, skipping deletion [e2e-llm-inference-service] CI namespace teardown complete [e2e-llm-inference-service] Creating namespace kserve-ci-e2e-test [e2e-llm-inference-service] namespace/kserve-ci-e2e-test created [e2e-llm-inference-service] Applying S3 artifact secret [e2e-llm-inference-service] secret/mlpipeline-s3-artifact created [e2e-llm-inference-service] Applying storage-config secret [e2e-llm-inference-service] secret/storage-config created [e2e-llm-inference-service] Applying SeaweedFS S3 credentials secret [e2e-llm-inference-service] secret/seaweedfs-s3-creds created [e2e-llm-inference-service] Linking seaweedfs-s3-creds to default service account [e2e-llm-inference-service] Creating odh-trusted-ca-bundle configmap [e2e-llm-inference-service] configmap/odh-trusted-ca-bundle created [e2e-llm-inference-service] Installing ServingRuntimes [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-autogluonserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-huggingfaceserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-huggingfaceserver-multinode created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-lgbserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-mlserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-paddleserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-pmmlserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-predictiveserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-sklearnserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-tensorflow-serving created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-torchserve created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-tritonserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-vllmserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-xgbserver created [e2e-llm-inference-service] CI namespace setup complete [e2e-llm-inference-service] Setup complete [e2e-llm-inference-service] === E2E cluster / operator summary === [e2e-llm-inference-service] Client Version: 4.20.11 [e2e-llm-inference-service] Kustomize Version: v5.6.0 [e2e-llm-inference-service] Server Version: 4.21.26 [e2e-llm-inference-service] Kubernetes Version: v1.34.9 [e2e-llm-inference-service] ClusterVersion desired: 4.21.26 [e2e-llm-inference-service] ClusterVersion history (latest): 4.21.26 (Completed) [e2e-llm-inference-service] CSVs in kuadrant-system: [e2e-llm-inference-service] authorino-operator.v1.4.2 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.0 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.1 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.1 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.2 Succeeded [e2e-llm-inference-service] CSVs in openshift-keda: [e2e-llm-inference-service] authorino-operator.v1.4.2 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.0 Succeeded [e2e-llm-inference-service] custom-metrics-autoscaler.v2.19.0-2 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.1 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.1 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.2 Succeeded [e2e-llm-inference-service] CSVs in cert-manager-operator: [e2e-llm-inference-service] authorino-operator.v1.4.2 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.0 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.1 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.1 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.2 Succeeded [e2e-llm-inference-service] CSVs in openshift-lws-operator: [e2e-llm-inference-service] authorino-operator.v1.4.2 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.0 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.1 Succeeded [e2e-llm-inference-service] leader-worker-set.v1.0.0 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.1 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.2 Succeeded [e2e-llm-inference-service] CSVs in openshift-operators (ODH / shared operators, filtered): [e2e-llm-inference-service] authorino-operator.v1.4.2 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.1 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.1 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.2 Succeeded [e2e-llm-inference-service] Kuadrant / Authorino (diagnostics): [e2e-llm-inference-service] CRD kuadrants.kuadrant.io versions: v1beta1 served=true storage=true [e2e-llm-inference-service] Subscriptions in kuadrant-system: [e2e-llm-inference-service] authorino-operator-stable-redhat-operators-openshift-marketplace stable redhat-operators authorino-operator.v1.4.2 [e2e-llm-inference-service] dns-operator-stable-redhat-operators-openshift-marketplace stable redhat-operators dns-operator.v1.4.1 [e2e-llm-inference-service] limitador-operator-stable-redhat-operators-openshift-marketplace stable redhat-operators limitador-operator.v1.4.1 [e2e-llm-inference-service] rhcl-operator stable redhat-operators rhcl-operator.v1.4.2 [e2e-llm-inference-service] Kuadrant CR conditions (kuadrant/kuadrant-system): [e2e-llm-inference-service] Ready=True (Ready) [e2e-llm-inference-service] KServe deployments in kserve: [e2e-llm-inference-service] kserve-controller-manager: ready=1 image=quay.io/opendatahub/kserve-controller@sha256:e84ef678972994166b9c11fc0192a03ca1e4fecf65c513bcc81ed8eba01106de [e2e-llm-inference-service] imageID: quay.io/opendatahub/kserve-controller@sha256:750b90ceba25d04b9693ea45013a80f1d721c58c47fdd3ed3deccac370d0764c [e2e-llm-inference-service] odh-model-controller: ready=1 image=quay.io/opendatahub/odh-model-controller:odh-incubating [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-model-controller@sha256:b5c835d640b650f164ba7f36d7008760c88e2974efa65853ee167323c441580c [e2e-llm-inference-service] llmisvc-controller-manager: ready=1 image=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:75646d30f65570d55e8a328e38a96abb4134409902fc2e94b8713dc206a6d886 [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:75646d30f65570d55e8a328e38a96abb4134409902fc2e94b8713dc206a6d886 [e2e-llm-inference-service] === End E2E cluster / operator summary === [e2e-llm-inference-service] /workspace/source [e2e-llm-inference-service] CA certificate extracted [e2e-llm-inference-service] REQUESTS_CA_BUNDLE=/tmp/ca.crt [e2e-llm-inference-service] Run E2E tests: llminferenceservice and cluster_cpu and not autoscaling and not tracing [e2e-llm-inference-service] Starting E2E functional tests ... [e2e-llm-inference-service] Parallelism requested for pytest is 2 [e2e-llm-inference-service] ============================= test session starts ============================== [e2e-llm-inference-service] platform linux -- Python 3.11.13, pytest-7.4.4, pluggy-1.5.0 -- /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] cachedir: .pytest_cache [e2e-llm-inference-service] metadata: {'Python': '3.11.13', 'Platform': 'Linux-5.14.0-570.124.1.el9_6.x86_64-x86_64-with-glibc2.34', 'Packages': {'pytest': '7.4.4', 'pluggy': '1.5.0'}, 'Plugins': {'xdist': '3.6.1', 'asyncio': '0.23.8', 'metadata': '3.1.1', 'cov': '5.0.0', 'httpx': '0.30.0', 'json-report': '1.5.0', 'anyio': '4.9.0'}, 'PLATFORM': 'el9'} [e2e-llm-inference-service] rootdir: /workspace/source/test/e2e [e2e-llm-inference-service] configfile: pytest.ini [e2e-llm-inference-service] plugins: xdist-3.6.1, asyncio-0.23.8, metadata-3.1.1, cov-5.0.0, httpx-0.30.0, json-report-1.5.0, anyio-4.9.0 [e2e-llm-inference-service] asyncio: mode=Mode.STRICT [e2e-llm-inference-service] created: 2/2 workers [e2e-llm-inference-service] 2 workers [73 items] [e2e-llm-inference-service] [e2e-llm-inference-service] scheduling tests via WorkStealingScheduling [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-utilization-detector] 2026-07-28 17:29:10.857 6480 kserve INFO [conftest.py:configure_logger():40] Logger configured [e2e-llm-inference-service] 2026-07-28 17:29:10.858 6477 kserve INFO [conftest.py:configure_logger():40] Logger configured [e2e-llm-inference-service] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-utilization-detector] [e2e-llm-inference-service] llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-concurrency-detector] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-concurrency-detector] [e2e-llm-inference-service] llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-with-section-name] 2026-07-28 17:31:27.589 6477 kserve.trace Checking Gateway router-gateway-1 in namespace e2e-test-gateway-section-name-propagation-49b5edd9 [e2e-llm-inference-service] 2026-07-28 17:31:27.589 6477 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-1 in namespace e2e-test-gateway-section-name-propagation-49b5edd9 [e2e-llm-inference-service] 2026-07-28 17:31:27.622 6477 kserve.trace Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-28 17:31:27.622 6477 kserve.trace INFO [gw_api.py:create_or_update_gateway():62] Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-28 17:31:27.629 6477 kserve.trace ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-28 17:31:27.629 6477 kserve.trace INFO [gw_api.py:create_or_update_gateway():70] ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-with-section-name] [e2e-llm-inference-service] llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-without-section-name] 2026-07-28 17:31:56.184 6477 kserve.trace Checking Gateway router-gateway-1 in namespace e2e-test-gateway-section-name-propagation-94799d44 [e2e-llm-inference-service] 2026-07-28 17:31:56.184 6477 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-1 in namespace e2e-test-gateway-section-name-propagation-94799d44 [e2e-llm-inference-service] 2026-07-28 17:31:56.215 6477 kserve.trace Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-28 17:31:56.215 6477 kserve.trace INFO [gw_api.py:create_or_update_gateway():62] Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-28 17:31:56.222 6477 kserve.trace ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-28 17:31:56.222 6477 kserve.trace INFO [gw_api.py:create_or_update_gateway():70] ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-without-section-name] [e2e-llm-inference-service] llmisvc/test_llm_auth.py::test_llm_auth_enabled_requires_token[cluster_cpu-cluster_single_node-auth-enabled-default] [e2e-llm-inference-service] [gw1] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] FAILED llmisvc/test_llm_auth.py::test_llm_auth_enabled_requires_token[cluster_cpu-cluster_single_node-auth-enabled-default] [e2e-llm-inference-service] llmisvc/test_llm_auth.py::test_llm_auth_invalid_token_rejected[cluster_cpu-cluster_single_node-auth-invalid-token] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-inline-config-workload-llmd-simulator] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-inline-config-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-qwen2.5-0.5b] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_auth.py::test_llm_auth_invalid_token_rejected[cluster_cpu-cluster_single_node-auth-invalid-token] [e2e-llm-inference-service] llmisvc/test_llm_auth.py::test_llm_auth_disabled_no_token_required[cluster_cpu-cluster_single_node-auth-disabled] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-qwen2.5-0.5b] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-configmap-ref-workload-llmd-simulator] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-configmap-ref-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-replicas-workload-llmd-simulator] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_auth.py::test_llm_auth_disabled_no_token_required[cluster_cpu-cluster_single_node-auth-disabled] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_hpa_deployment[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-replicas-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-custom-template-workload-llmd-simulator] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-custom-template-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-nonzero-threshold-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-nonzero-threshold-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-tokenizer-kvcache-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-tokenizer-kvcache-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator0] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator1] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator2] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator2] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf0] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] [e2e-llm-inference-service] [gw0] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_hpa_deployment[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_keda_deployment[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-keda] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-pvc] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-pvc] [e2e-llm-inference-service] [gw0] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_keda_deployment[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-keda] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_hpa_lws[cluster_cpu-cluster_multi_node-router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] [gw1] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-pvc] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service_config_deletion.py::test_config_finalizer_added [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_config_deletion.py::test_config_finalizer_added [e2e-llm-inference-service] llmisvc/test_llm_inference_service_config_deletion.py::test_config_deletion_blocked_when_referenced [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_config_deletion.py::test_config_deletion_blocked_when_referenced [e2e-llm-inference-service] llmisvc/test_llm_inference_service_config_deletion.py::test_config_deletion_allowed_when_unreferenced [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_config_deletion.py::test_config_deletion_allowed_when_unreferenced [e2e-llm-inference-service] llmisvc/test_llm_inference_service_config_deletion.py::test_config_deletion_unblocked_after_service_deleted [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_config_deletion.py::test_config_deletion_unblocked_after_service_deleted [e2e-llm-inference-service] llmisvc/test_llm_inference_service_config_deletion.py::test_well_known_config_deletion_prevented_by_webhook [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_config_deletion.py::test_well_known_config_deletion_prevented_by_webhook [e2e-llm-inference-service] llmisvc/test_llm_inference_service_config_deletion.py::test_well_known_config_deletion_blocked_by_implicit_reference [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_config_deletion.py::test_well_known_config_deletion_blocked_by_implicit_reference [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha1_to_v1alpha2_conversion [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha1_to_v1alpha2_conversion [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha2_to_v1alpha1_conversion [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha2_to_v1alpha1_conversion [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_criticality_preservation_via_annotations [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_criticality_preservation_via_annotations [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_lora_criticality_preservation [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_lora_criticality_preservation [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_round_trip_conversion_preserves_fields [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_round_trip_conversion_preserves_fields [e2e-llm-inference-service] llmisvc/test_llm_inference_service_stop.py::test_llm_stop_feature[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_stop.py::test_llm_stop_feature[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-single-lora-adapter-hf] [e2e-llm-inference-service] [gw0] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_hpa_lws[cluster_cpu-cluster_multi_node-router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_keda_lws[cluster_cpu-cluster_multi_node-router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-keda] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-single-lora-adapter-hf] [e2e-llm-inference-service] llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-multiple-lora-adapters] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-multiple-lora-adapters] [e2e-llm-inference-service] llmisvc/test_llm_tls.py::test_llm_tls_resources[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_tls.py::test_llm_tls_resources[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_prestop_hook.py::test_prestop_hook[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_prestop_hook.py::test_prestop_hook[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_rolling_upgrade.py::test_rolling_upgrade_coordination[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_rolling_upgrade.py::test_rolling_upgrade_coordination[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_storage_version_migration.py::TestStorageVersionMigration::test_storage_version_migration_after_simulated_upgrade [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_storage_version_migration.py::TestStorageVersionMigration::test_storage_version_migration_after_simulated_upgrade [e2e-llm-inference-service] [gw0] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_keda_lws[cluster_cpu-cluster_multi_node-router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-keda] [e2e-llm-inference-service] [e2e-llm-inference-service] =================================== FAILURES =================================== [e2e-llm-inference-service] _ test_llm_inference_service[router-no-scheduler-workload-single-cpu-model-fb-opt-125m] _ [e2e-llm-inference-service] [gw1] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-no-scheduler', 'workload-single-cpu', 'model-fb-opt-125m'], prompt='What is KServe?', serv... {'name': 'model-fb-opt-125m-llmisvc-model-7a2ca70d'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-gateway-ref", [e2e-llm-inference-service] "router-with-managed-route", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] expected_gateway="router-gateway-1", [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda tc: create_router_resources( [e2e-llm-inference-service] gateways=[ [e2e-llm-inference-service] make_router_gateway( [e2e-llm-inference-service] "router-gateway-1", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="custom-route-timeout-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="router-with-refs-test", [e2e-llm-inference-service] expected_gateway="router-gateway-1", [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda tc: create_router_resources( [e2e-llm-inference-service] gateways=[ [e2e-llm-inference-service] make_router_gateway( [e2e-llm-inference-service] "router-gateway-1", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] routes=[ [e2e-llm-inference-service] make_router_main_route( [e2e-llm-inference-service] "router-route-1", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] "router-gateway-1", [e2e-llm-inference-service] "router-with-refs-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] make_router_health_route( [e2e-llm-inference-service] "router-route-2", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] "router-gateway-1", [e2e-llm-inference-service] "router-with-refs-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=["router-managed", "workload-pd-cpu", "model-fb-opt-125m"], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="custom-route-timeout-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="router-with-refs-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] expected_gateway="router-gateway-2", [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda tc: create_router_resources( [e2e-llm-inference-service] gateways=[ [e2e-llm-inference-service] make_router_gateway( [e2e-llm-inference-service] "router-gateway-2", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] routes=[ [e2e-llm-inference-service] make_router_main_route( [e2e-llm-inference-service] "router-route-3", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] "router-gateway-2", [e2e-llm-inference-service] "router-with-refs-pd-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] make_router_health_route( [e2e-llm-inference-service] "router-route-4", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] "router-gateway-2", [e2e-llm-inference-service] "router-with-refs-pd-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-dp-ep-gpu", [e2e-llm-inference-service] "workload-dp-ep-prefill-gpu", [e2e-llm-inference-service] "model-deepseek-v2-lite", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="Delve into the multifaceted implications of a fully disaggregated cloud architecture, specifically " [e2e-llm-inference-service] "where the compute plane (P) and the data plane (D) are independently deployed and managed for a " [e2e-llm-inference-service] "geographically distributed, high-throughput, low-latency microservices ecosystem. Beyond the " [e2e-llm-inference-service] "fundamental challenges of network latency and data consistency, elaborate on the advanced " [e2e-llm-inference-service] "considerations and trade-offs inherent in such a setup: 1. Network Architecture and Protocols: " [e2e-llm-inference-service] "How would the network fabric and underlying protocols (e.g., RDMA, custom transport layers) need to " [e2e-llm-inference-service] "evolve to support optimal performance and minimize inter-plane communication overhead, especially for " [e2e-llm-inference-service] "synchronous operations? Discuss the role of network programmability (e.g., SDN, P4) in dynamically " [e2e-llm-inference-service] "optimizing routing and traffic flow between P and D. 2. Advanced Data Consistency and Durability: " [e2e-llm-inference-service] "Explore sophisticated data consistency models (e.g., causal consistency, strong eventual consistency) " [e2e-llm-inference-service] "and their applicability in balancing performance and data integrity across a globally distributed data plane. " [e2e-llm-inference-service] "Detail strategies for ensuring data durability and fault tolerance, including multi-region replication, " [e2e-llm-inference-service] "intelligent partitioning, and recovery mechanisms in the event of partial or full plane failures. " [e2e-llm-inference-service] "3. Dynamic Resource Orchestration and Cost Optimization: Analyze how an orchestration layer would intelligently " [e2e-llm-inference-service] "manage the independent scaling of compute (P) and data (D) resources, considering fluctuating workloads, " [e2e-llm-inference-service] "cost efficiency, and performance targets (e.g., using predictive analytics for resource provisioning). " [e2e-llm-inference-service] "Discuss mechanisms for dynamically reallocating compute nodes to different data partitions based on " [e2e-llm-inference-service] "workload patterns and data locality, potentially involving live migration strategies. " [e2e-llm-inference-service] "4. Security and Compliance in a Distributed Landscape: Address the enhanced security perimeter " [e2e-llm-inference-service] "challenges, including securing communication channels between P and D (encryption in transit, mutual TLS), " [e2e-llm-inference-service] "fine-grained access control to data at rest and in motion, and identity management across disaggregated " [e2e-llm-inference-service] "components. Discuss how such an architecture impacts compliance with regulatory frameworks (e.g., GDPR, HIPAA) " [e2e-llm-inference-service] "concerning data sovereignty, privacy, and auditability. 5. Operational Complexity and Observability: " [e2e-llm-inference-service] "Examine the increased complexity in monitoring, logging, and tracing across highly decoupled compute and " [e2e-llm-inference-service] "data planes. What specialized tooling and practices (e.g., distributed tracing with OpenTelemetry, advanced AIOps) " [e2e-llm-inference-service] "would be essential? How would incident response and troubleshooting differ in this disaggregated environment " [e2e-llm-inference-service] "compared to traditional integrated systems? Consider the challenges of pinpointing root causes across " [e2e-llm-inference-service] "independent failures. 6. Real-world Applicability and Future Trends: Identify specific industries " [e2e-llm-inference-service] "or use cases (e.g., high-frequency trading, IoT edge processing, large language model inference) " [e2e-llm-inference-service] "where the benefits of P/D disaggregation would strongly outweigh its complexities. " [e2e-llm-inference-service] "Conclude by speculating on emerging technologies or paradigms (e.g., serverless compute functions " [e2e-llm-inference-service] "directly interacting with object storage, in-memory disaggregation) that could further drive or " [e2e-llm-inference-service] "transform P/D disaggregation in cloud computing.", [e2e-llm-inference-service] max_tokens=2000, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_gpu, [e2e-llm-inference-service] pytest.mark.cluster_nvidia, [e2e-llm-inference-service] pytest.mark.cluster_nvidia_roce, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-no-scheduler", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.no_scheduler, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="This test simulates DP+EP that can run on CPU, the idea is to test the LWS-based deployment, " [e2e-llm-inference-service] "but without the resources requirements for DP+EP (GPUs and ROCe/IB).", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_multi_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler config tests [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-inline-config-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Chat completions endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] model_name="Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-configmap-ref", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-configmap-ref-test", [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda tc: create_scheduler_configmap(namespace=tc.namespace) [e2e-llm-inference-service] ], [e2e-llm-inference-service] after_test=[ [e2e-llm-inference-service] lambda tc: delete_scheduler_configmap(namespace=tc.namespace) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-replicas", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-ha-replicas-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-custom-template", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-custom-template-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler v0.6 → v0.7 migration tests. [e2e-llm-inference-service] # Deploy v0.6-style configs and verify the controller migrates them [e2e-llm-inference-service] # so the v0.7 scheduler boots successfully. [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-v06-pd-config-migration", [e2e-llm-inference-service] "workload-llmd-simulator-pd", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-v06-pd-migration-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-v06-nonzero-threshold-migration", [e2e-llm-inference-service] "workload-llmd-simulator-pd", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-v06-threshold-migration-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Standalone tokenizer — clean path: token-producer in inline config [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-tokenizer-kvcache", [e2e-llm-inference-service] "workload-llmd-simulator-kvcache", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="tokenizer-clean-path-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Standalone tokenizer — migration path: legacy precise-prefix-cache-scorer [e2e-llm-inference-service] # triggers auto-provisioned tokenizer without explicit tokenizer:{} field [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-precise-prefix-cache-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator-kvcache", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="tokenizer-migration-path-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Models endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="data"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/chat/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — LoRA adapter [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] model_name="publishers/{namespace}/models/lora-adapter-1", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "publishers/{namespace}/models/lora-adapter-1" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/lora-adapter-1", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/models (base + LoRA) [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=assert_models_contains( [e2e-llm-inference-service] "facebook/opt-125m", [e2e-llm-inference-service] "publishers/{namespace}/models/facebook/opt-125m", [e2e-llm-inference-service] "lora-adapter-1", [e2e-llm-inference-service] "publishers/{namespace}/models/lora-adapter-1", [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # PVC storage tests -- validate direct PVC volume mount with real vLLM serving [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] before_test=[lambda tc: ensure_pvc_with_model(namespace=tc.namespace)], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] before_test=[lambda tc: ensure_pvc_with_model(namespace=tc.namespace)], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] before_test=[lambda tc: ensure_pvc_with_model(namespace=tc.namespace)], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_multi_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_inference_service(test_case: TestCase): # noqa: F811 [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = KServeClient( [e2e-llm-inference-service] config_file=os.environ.get("KUBECONFIG", "~/.kube/config"), [e2e-llm-inference-service] client_configuration=client.Configuration(), [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] prefix = test_case.log_prefix [e2e-llm-inference-service] [e2e-llm-inference-service] test_failed = False [e2e-llm-inference-service] try: [e2e-llm-inference-service] print(f"{prefix} Creating LLMInferenceService {service_name}") [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] print(f"{prefix} Waiting for LLMInferenceService {service_name} to be ready") [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:866: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...c-mod-5c1136e5'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-llmisvc-model-7a2ca70d'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-07-28T17:29:11.637457', start_time = 1785259751.6377478 [e2e-llm-inference-service] duration = 900.5412843227386, timestamp_end = '2026-07-28T17:44:12.179034' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....llmisvc-mod-5c1136e5'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-llmisvc-model-7a2ca70d'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1376: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7ff123bdd440> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1387: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T17:29:44Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T17:30:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:29:44Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T17:30:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T17:29:44Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T17:30:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1371: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO kserve:conftest.py:40 Logger configured [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:82 Created test namespace e2e-test-llm-inference-service-fac1c4ae [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret seaweedfs-s3-creds from kserve-ci-e2e-test to e2e-test-llm-inference-service-fac1c4ae [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret storage-config from kserve-ci-e2e-test to e2e-test-llm-inference-service-fac1c4ae [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:178 Patched default SA in e2e-test-llm-inference-service-fac1c4ae with secret seaweedfs-s3-creds [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:162 ConfigMap odh-kserve-custom-ca-bundle already exists in e2e-test-llm-inference-service-fac1c4ae [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:159 Copied ConfigMap odh-trusted-ca-bundle from kserve-ci-e2e-test to e2e-test-llm-inference-service-fac1c4ae [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig router-no-scheduler-llmisvc-mod-ff50d279 in namespace e2e-test-llm-inference-service-fac1c4ae [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig router-no-scheduler-llmisvc-mod-ff50d279 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig router-no-scheduler-llmisvc-mod-ff50d279 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig workload-single-cpu-llmisvc-mod-5c1136e5 in namespace e2e-test-llm-inference-service-fac1c4ae [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig workload-single-cpu-llmisvc-mod-5c1136e5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig workload-single-cpu-llmisvc-mod-5c1136e5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig model-fb-opt-125m-llmisvc-model-7a2ca70d in namespace e2e-test-llm-inference-service-fac1c4ae [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig model-fb-opt-125m-llmisvc-model-7a2ca70d [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig model-fb-opt-125m-llmisvc-model-7a2ca70d [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:91 Created ConfigMap kserve-ci-e2e-test/gateway-proxy-config [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_inference_service] [2026-07-28T17:29:11.392490] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-no-scheduler', 'workload-single-cpu', 'model-fb-opt-125m'], prompt='What is KServe?', service_name='llmisvc-model-fb-opt-125m-route-3c960099', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, namespace='e2e-test-llm-inference-service-fac1c4ae', before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-route-3c960099', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-inference-service-fac1c4ae', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-no-scheduler-llmisvc-mod-ff50d279'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-5c1136e5'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-llmisvc-model-7a2ca70d'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-28T17:29:11.406629] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-route-3c960099', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-inference-service-fac1c4ae', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-no-scheduler-llmisvc-mod-ff50d279'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-5c1136e5'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-llmisvc-model-7a2ca70d'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-28T17:29:11.637314] end - ✅ in 0.230s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-28T17:29:11.637457] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-route-3c960099', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-inference-service-fac1c4ae', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-no-scheduler-llmisvc-mod-ff50d279'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-5c1136e5'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-llmisvc-model-7a2ca70d'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T17:29:44Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T17:29:44Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:29:44Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T17:29:44Z', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T17:29:44Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T17:29:44Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T17:29:44Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T17:30:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:29:44Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T17:30:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T17:29:44Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T17:30:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1391 Timed out waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T17:29:44Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T17:30:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:29:44Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T17:30:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T17:29:44Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T17:30:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-07-28T17:44:12.179034] end - ❌ 900.541s: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T17:29:44Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T17:30:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:29:44Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T17:30:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T17:29:44Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T17:30:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:903 [router-no-scheduler-workload-single-cpu-model-fb-opt-125m] ❌ ERROR: Failed to call llm inference service llmisvc-model-fb-opt-125m-route-3c960099: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T17:29:44Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T17:30:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:29:44Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T17:30:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T17:29:44Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T17:30:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:243 # Diagnostics for 'llmisvc-model-fb-opt-125m-route-3c960099' in 'e2e-test-llm-inference-service-fac1c4ae' [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:244 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:247 # LLMInferenceService llmisvc-model-fb-opt-125m-route-3c960099 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:256 apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] security.opendatahub.io/enable-auth: 'false' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:29:11Z' [e2e-llm-inference-service] finalizers: [e2e-llm-inference-service] - serving.kserve.io/llmisvc-finalizer [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:security.opendatahub.io/enable-auth: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:baseRefs: {} [e2e-llm-inference-service] manager: OpenAPI-Generator [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T17:29:11Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:finalizers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] v:"serving.kserve.io/llmisvc-finalizer": {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T17:29:19Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:addresses: {} [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-router-route: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-scheduler: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-scheduler-latency-predictor: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-tokenizer: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-tracing: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-worker-data-parallel: {} [e2e-llm-inference-service] f:appliedConfigs: {} [e2e-llm-inference-service] f:conditions: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:router: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:gateways: {} [e2e-llm-inference-service] f:url: {} [e2e-llm-inference-service] f:workloads: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:primary: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-28T17:30:03Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099 [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-fac1c4ae [e2e-llm-inference-service] resourceVersion: '22677' [e2e-llm-inference-service] uid: 5c6f11ac-adba-4e63-8af1-189e8f5b3f44 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] baseRefs: [e2e-llm-inference-service] - name: router-no-scheduler-llmisvc-mod-ff50d279 [e2e-llm-inference-service] - name: workload-single-cpu-llmisvc-mod-5c1136e5 [e2e-llm-inference-service] - name: model-fb-opt-125m-llmisvc-model-7a2ca70d [e2e-llm-inference-service] model: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099 [e2e-llm-inference-service] uri: '' [e2e-llm-inference-service] status: [e2e-llm-inference-service] addresses: [e2e-llm-inference-service] - name: gateway-external-model-routing [e2e-llm-inference-service] url: http://a803f1ae9e25d4fc48c1c4d8072319d1-1138438229.us-east-1.elb.amazonaws.com/ [e2e-llm-inference-service] - name: gateway-external [e2e-llm-inference-service] url: http://a803f1ae9e25d4fc48c1c4d8072319d1-1138438229.us-east-1.elb.amazonaws.com/e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099 [e2e-llm-inference-service] - name: gateway-external [e2e-llm-inference-service] url: http://a803f1ae9e25d4fc48c1c4d8072319d1-1138438229.us-east-1.elb.amazonaws.com/publishers/e2e-test-llm-inference-service-fac1c4ae/models/facebook/opt-125m [e2e-llm-inference-service] - name: gateway-internal-model-routing [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/ [e2e-llm-inference-service] - name: gateway-internal [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099 [e2e-llm-inference-service] - name: gateway-internal [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/publishers/e2e-test-llm-inference-service-fac1c4ae/models/facebook/opt-125m [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-template: kserve-config-llm-decode-template [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-worker-data-parallel: kserve-config-llm-decode-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-template: kserve-config-llm-prefill-template [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-worker-data-parallel: kserve-config-llm-prefill-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-router-route: kserve-config-llm-router-route [e2e-llm-inference-service] serving.kserve.io/config-llm-scheduler: kserve-config-llm-scheduler [e2e-llm-inference-service] serving.kserve.io/config-llm-scheduler-latency-predictor: kserve-config-llm-scheduler-latency-predictor [e2e-llm-inference-service] serving.kserve.io/config-llm-template: kserve-config-llm-template [e2e-llm-inference-service] serving.kserve.io/config-llm-tokenizer: kserve-config-llm-tokenizer [e2e-llm-inference-service] serving.kserve.io/config-llm-tracing: kserve-config-llm-tracing [e2e-llm-inference-service] serving.kserve.io/config-llm-worker-data-parallel: kserve-config-llm-worker-data-parallel [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T17:29:44Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: HTTPRoutesReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T17:30:03Z' [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: MainWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T17:29:44Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: PresetsCombined [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T17:30:03Z' [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: Ready [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T17:29:44Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: RouterReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T17:30:03Z' [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: WorkloadsReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] url: http://a803f1ae9e25d4fc48c1c4d8072319d1-1138438229.us-east-1.elb.amazonaws.com/e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:68 TIME NAMESPACE SOURCE TYPE REASON MESSAGE [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:69 -------------------------------------------------------------------------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 17:29:43 e2e-test-llm-inference-service-fac1c4ae replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-3c960099-kserve-788d7df956lftbb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None e2e-test-llm-inference-service-fac1c4ae Normal Scheduled Successfully assigned e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-788d7df956lftbb to ip-10-0-135-20.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 17:29:43 e2e-test-llm-inference-service-fac1c4ae multus Normal AddedInterface Add eth0 [10.134.0.25/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 17:29:43 e2e-test-llm-inference-service-fac1c4ae kubelet/ip-10-0-135-20.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 17:29:44 e2e-test-llm-inference-service-fac1c4ae kubelet/ip-10-0-135-20.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 17:29:44 e2e-test-llm-inference-service-fac1c4ae kubelet/ip-10-0-135-20.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 17:29:43 e2e-test-llm-inference-service-fac1c4ae deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-3c960099-kserve-788d7df956 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 17:29:42 e2e-test-llm-inference-service-fac1c4ae OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-3c960099-kserve-route-authn: failed to get HTTPRoute e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-3c960099-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 17:29:43 e2e-test-llm-inference-service-fac1c4ae LLMInferenceServiceController Normal Created Created v1.Secret e2e-test-llm-inference-service-fac1c4ae/llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 17:29:43 e2e-test-llm-inference-service-fac1c4ae LLMInferenceServiceController Normal Created Created v1.Deployment e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 17:29:43 e2e-test-llm-inference-service-fac1c4ae LLMInferenceServiceController Normal Created Created v1.Service e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 17:29:43 e2e-test-llm-inference-service-fac1c4ae LLMInferenceServiceController Normal Created Created v1.HTTPRoute e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 17:29:44 e2e-test-llm-inference-service-fac1c4ae LLMInferenceServiceController Normal Created Created v1.DestinationRule e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 17:30:03 e2e-test-llm-inference-service-fac1c4ae LLMInferenceServiceController Normal Updated Updated v1.Secret e2e-test-llm-inference-service-fac1c4ae/llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:172 ### Pod llmisvc-model-fb-opt-125m-route-3c960099-kserve-788d7df956lftbb (phase=Pending) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:212 #### init-container 'storage-initializer' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:225 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:226 2026-07-28 17:29:44.246 1 storage.initializer INFO [initializer-entrypoint:():17] Initializing, args: (src_uri, dest_path): [('hf://facebook/opt-125m', '/mnt/models')] [e2e-llm-inference-service] 2026-07-28 17:29:44.247 1 storage.initializer INFO [kserve_storage.py:download():168] Copying contents of hf://facebook/opt-125m to local [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/wPaCkH-WbT7GsmxMKKrNZTV4nSM=.ac481c8eb05e4d2496fbe076a38a7b4835dd733d.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_db906f7f-94a1-48f6-b1ca-6a5797d2ce9c'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/5HHJ6px3_ZRDOG3OxNZMhuycwOk=.a591333512516f58bf2002045dece909a0ccdb8b.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_00018b3f-7383-464f-9980-c1c5a919f621'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/Xn7B-BWUGOee2Y6hCZtEhtFu4BE=.38c05904caf6e5b9f04ecda5c973d77e6c1da151.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_cd257bcc-2904-416a-acca-b61dbfdbe5e6'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/8_PA_wEVGiVa2goH2H4KQOQpvVY=.b3fb716a3024261980becb2382e31a3780985130.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_426804ba-3491-45c5-a258-c1d75702caaa'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:212 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:228 # -- logs (current): unavailable ((400) [e2e-llm-inference-service] Reason: Bad Request [e2e-llm-inference-service] HTTP response headers: HTTPHeaderDict({'Audit-Id': 'ce5e3c4e-578d-40fa-ad53-5c1a07ae31f6', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'Date': 'Tue, 28 Jul 2026 17:44:12 GMT', 'Content-Length': '245'}) [e2e-llm-inference-service] HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"container \"main\" in pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-788d7df956lftbb\" is waiting to start: PodInitializing","reason":"BadRequest","code":400} [e2e-llm-inference-service] [e2e-llm-inference-service] ) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-fac1c4ae [e2e-llm-inference-service] uid: 7ba18ff2-e19f-4e38-824d-e109c657a919 [e2e-llm-inference-service] resourceVersion: '22367' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:29:43Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-3c960099 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-28T17:29:43Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T17:29:44Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - notReadyAddresses: [e2e-llm-inference-service] - ip: 10.134.0.25 [e2e-llm-inference-service] nodeName: ip-10-0-135-20.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-fac1c4ae [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-788d7df956lftbb [e2e-llm-inference-service] uid: ec8c770f-4829-4d70-809d-2d31b8297a9a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-788d7df956lftbb [e2e-llm-inference-service] generateName: llmisvc-model-fb-opt-125m-route-3c960099-kserve-788d7df956- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-fac1c4ae [e2e-llm-inference-service] uid: ec8c770f-4829-4d70-809d-2d31b8297a9a [e2e-llm-inference-service] resourceVersion: '22364' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:29:43Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-3c960099 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: 788d7df956 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.134.0.25/23"],"mac_address":"0a:58:0a:86:00:19","gateway_ips":["10.134.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.134.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.134.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.134.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.134.0.1"}],"ip_address":"10.134.0.25/23","gateway_ip":"10.134.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.134.0.25\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:86:00:19\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-788d7df956 [e2e-llm-inference-service] uid: 5367ca94-d638-4478-95fc-880caf0745db [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-135-20 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T17:29:43Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T17:29:43Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"5367ca94-d638-4478-95fc-880caf0745db"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_HIGH_PERFORMANCE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TOKIO_WORKER_THREADS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T17:29:43Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T17:29:44Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:initContainerStatuses: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.134.0.25"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kube-api-access-72cnn [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da [e2e-llm-inference-service] args: [e2e-llm-inference-service] - hf://facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: TOKIO_WORKER_THREADS [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: HF_HUB_ENABLE_HF_TRANSFER [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_HIGH_PERFORMANCE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_NUM_CONCURRENT_RANGE_GETS [e2e-llm-inference-service] value: '8' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-72cnn [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000710000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to infer\ [e2e-llm-inference-service] \ RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null\n\ [e2e-llm-inference-service] \ grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/* 2>/dev/null\n\ [e2e-llm-inference-service] \n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"$hca_dir\"\ [e2e-llm-inference-service] \ ]; then\n hca_name=$(basename \"$hca_dir\")\n port_state_file=\"\ [e2e-llm-inference-service] $hca_dir/ports/1/state\" # Assume port 1\n type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\ [e2e-llm-inference-service] \n\n echo \"[Infer RoCE] Check if the port state file ${port_state_file}\ [e2e-llm-inference-service] \ exists and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] &&\ [e2e-llm-inference-service] \ grep -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found active\ [e2e-llm-inference-service] \ HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n else\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Skipping inactive or down HCA: $hca_name\"\ [e2e-llm-inference-service] \n fi\n fi\n done\n\n # Check if we found any active HCAs\n\ [e2e-llm-inference-service] \ if [ ${#active_hcas[@]} -gt 0 ]; then\n # Join the array elements with\ [e2e-llm-inference-service] \ a comma\n hca_port_pairs=()\n for hca in \"${active_hcas[@]}\";\ [e2e-llm-inference-service] \ do\n hca_port_pairs+=(\"${hca}:1\")\n done\n\n active_hca_list=$(IFS=,;\ [e2e-llm-inference-service] \ echo \"${active_hcas[*]}\")\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\"\ [e2e-llm-inference-service] )\n echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n\ [e2e-llm-inference-service] \ export UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found. NCCL_IB_HCA\ [e2e-llm-inference-service] \ will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Finding GID_INDEX for each active HCA (SR-IOV compatible)...\"\ [e2e-llm-inference-service] \n\n # For SR-IOV environments, find the most common IPv4 RoCE v2 GID index\ [e2e-llm-inference-service] \ across all HCAs\n declare -A gid_index_count\n declare -A hca_gid_index\n\ [e2e-llm-inference-service] \n for hca_name in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Processing HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for\ [e2e-llm-inference-service] \ this HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"$tpath\"\ [e2e-llm-inference-service] \ 2>/dev/null; then\n idx=$(basename \"$tpath\")\n \ [e2e-llm-inference-service] \ gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n \ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo \"\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Found IPv4 RoCE v2 GID for ${hca_name}:\ [e2e-llm-inference-service] \ index=${idx}, gid=${gid_value}\"\n hca_gid_index[\"${hca_name}\"\ [e2e-llm-inference-service] ]=\"${idx}\"\n gid_index_count[\"${idx}\"]=$((${gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]} + 1))\n break # Use first found IPv4 GID per\ [e2e-llm-inference-service] \ HCA\n fi\n fi\n done\n done\n\n\ [e2e-llm-inference-service] \ # Find the most common GID index (most likely to be consistent across\ [e2e-llm-inference-service] \ nodes)\n best_gid_index=\"\"\n max_count=0\n for idx in \"\ [e2e-llm-inference-service] ${!gid_index_count[@]}\"; do\n count=${gid_index_count[\"${idx}\"]}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n \ [e2e-llm-inference-service] \ if [ $count -gt $max_count ]; then\n max_count=$count\n\ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n #\ [e2e-llm-inference-service] \ Use deterministic fallback if tied - prefer index 3 (SR-IOV standard)\n \ [e2e-llm-inference-service] \ if [ ${#gid_index_count[@]} -gt 1 ]; then\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Multiple GID indices found, selecting most common: ${best_gid_index}\"\n \ [e2e-llm-inference-service] \ # If there's a tie, prefer index 3 as it's most common in SR-IOV setups\n\ [e2e-llm-inference-service] \ if [ -n \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\"\ [e2e-llm-inference-service] \ -eq \"$max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for NCCL,\ [e2e-llm-inference-service] \ NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR: No valid\ [e2e-llm-inference-service] \ IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any HCA.\"\n \ [e2e-llm-inference-service] \ fi\n else\n echo \"[Infer RoCE] No active HCAs found, skipping GID_INDEX\ [e2e-llm-inference-service] \ inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints landed in vLLM\ [e2e-llm-inference-service] \ 0.16.0 (vllm-project/vllm#30011).\n# Older versions still need the blanket\ [e2e-llm-inference-service] \ --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+ ]] &&\ [e2e-llm-inference-service] \ [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort -V | head\ [e2e-llm-inference-service] \ -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout 40\"\ [e2e-llm-inference-service] \nfi\n\n# --kv-transfer-config with OffloadingConnector requires vLLM 0.22.0+\ [e2e-llm-inference-service] \ (vllm-project/vllm#40020).\nKV_TRANSFER_ARGS=\"\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.22.0\" ]; then\n if [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\" != *\"\ [e2e-llm-inference-service] --kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"* ]] &&\ [e2e-llm-inference-service] \ [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n KV_TRANSFER_ARGS=\"\"\ [e2e-llm-inference-service] \n fi\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name \"\ [e2e-llm-inference-service] facebook/opt-125m\" \"publishers/e2e-test-llm-inference-service-fac1c4ae/models/facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8000 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n ${KV_TRANSFER_ARGS} \\\n \\\n --enable-ssl-refresh \\\n --ssl-certfile\ [e2e-llm-inference-service] \ /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-72cnn [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000710000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: default [e2e-llm-inference-service] serviceAccount: default [e2e-llm-inference-service] nodeName: ip-10-0-135-20.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c27,c4 [e2e-llm-inference-service] fsGroup: 1000710000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-65t55 [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Pending [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T17:29:44Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T17:29:43Z' [e2e-llm-inference-service] reason: ContainersNotInitialized [e2e-llm-inference-service] message: 'containers with incomplete status: [storage-initializer]' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T17:29:43Z' [e2e-llm-inference-service] reason: ContainersNotReady [e2e-llm-inference-service] message: 'containers with unready status: [main]' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T17:29:43Z' [e2e-llm-inference-service] reason: ContainersNotReady [e2e-llm-inference-service] message: 'containers with unready status: [main]' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T17:29:43Z' [e2e-llm-inference-service] hostIP: 10.0.135.20 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.135.20 [e2e-llm-inference-service] podIP: 10.134.0.25 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.134.0.25 [e2e-llm-inference-service] startTime: '2026-07-28T17:29:43Z' [e2e-llm-inference-service] initContainerStatuses: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-07-28T17:29:44Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: false [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da [e2e-llm-inference-service] imageID: quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da [e2e-llm-inference-service] containerID: cri-o://4fc88d8b9aac6954b199294486888fd60ed0dece9cc27b881e8bf4694e780934 [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-72cnn [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000710000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000710000 [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] waiting: [e2e-llm-inference-service] reason: PodInitializing [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: false [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] imageID: '' [e2e-llm-inference-service] started: false [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-72cnn [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-fac1c4ae [e2e-llm-inference-service] uid: ce01253e-1e45-4bb3-af6d-6d10bb9b4ffb [e2e-llm-inference-service] resourceVersion: '22285' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:29:43Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-3c960099 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099 [e2e-llm-inference-service] uid: 5c6f11ac-adba-4e63-8af1-189e8f5b3f44 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T17:29:43Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"5c6f11ac-adba-4e63-8af1-189e8f5b3f44"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] targetPort: 8000 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-3c960099 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] clusterIP: 172.31.111.216 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.111.216 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-fac1c4ae [e2e-llm-inference-service] uid: 9bda3219-9d3b-4b61-8d9c-16732aaca7b5 [e2e-llm-inference-service] resourceVersion: '30206' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:29:43Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-3c960099 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099 [e2e-llm-inference-service] uid: 5c6f11ac-adba-4e63-8af1-189e8f5b3f44 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T17:29:43Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"5c6f11ac-adba-4e63-8af1-189e8f5b3f44"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:rollingUpdate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:maxSurge: {} [e2e-llm-inference-service] f:maxUnavailable: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_HIGH_PERFORMANCE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TOKIO_WORKER_THREADS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T17:39:44Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:unavailableReplicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-3c960099 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-3c960099 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da [e2e-llm-inference-service] args: [e2e-llm-inference-service] - hf://facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: TOKIO_WORKER_THREADS [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: HF_HUB_ENABLE_HF_TRANSFER [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_HIGH_PERFORMANCE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_NUM_CONCURRENT_RANGE_GETS [e2e-llm-inference-service] value: '8' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n # Check if\ [e2e-llm-inference-service] \ we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ # Join the array elements with a comma\n hca_port_pairs=()\n \ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n hca_port_pairs+=(\"\ [e2e-llm-inference-service] ${hca}:1\")\n done\n\n active_hca_list=$(IFS=,; echo \"${active_hcas[*]}\"\ [e2e-llm-inference-service] )\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\n# --kv-transfer-config with OffloadingConnector requires vLLM\ [e2e-llm-inference-service] \ 0.22.0+ (vllm-project/vllm#40020).\nKV_TRANSFER_ARGS=\"\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.22.0\" ]; then\n if [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\" !=\ [e2e-llm-inference-service] \ *\"--kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"\ [e2e-llm-inference-service] * ]] && [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n KV_TRANSFER_ARGS=\"\ [e2e-llm-inference-service] \"\n fi\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \"publishers/e2e-test-llm-inference-service-fac1c4ae/models/facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8000 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n ${KV_TRANSFER_ARGS} \\\n \\\n --enable-ssl-refresh \\\n --ssl-certfile\ [e2e-llm-inference-service] \ /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: RollingUpdate [e2e-llm-inference-service] rollingUpdate: [e2e-llm-inference-service] maxUnavailable: 25% [e2e-llm-inference-service] maxSurge: 25% [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] unavailableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastUpdateTime: '2026-07-28T17:29:43Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T17:29:43Z' [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastUpdateTime: '2026-07-28T17:39:44Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T17:39:44Z' [e2e-llm-inference-service] reason: ProgressDeadlineExceeded [e2e-llm-inference-service] message: ReplicaSet "llmisvc-model-fb-opt-125m-route-3c960099-kserve-788d7df956" [e2e-llm-inference-service] has timed out progressing. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-788d7df956 [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-fac1c4ae [e2e-llm-inference-service] uid: 5367ca94-d638-4478-95fc-880caf0745db [e2e-llm-inference-service] resourceVersion: '22296' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:29:43Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-3c960099 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: 788d7df956 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '2' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve [e2e-llm-inference-service] uid: 9bda3219-9d3b-4b61-8d9c-16732aaca7b5 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T17:29:43Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"9bda3219-9d3b-4b61-8d9c-16732aaca7b5"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_HIGH_PERFORMANCE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TOKIO_WORKER_THREADS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T17:29:43Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-3c960099 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: 788d7df956 [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-3c960099 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: 788d7df956 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da [e2e-llm-inference-service] args: [e2e-llm-inference-service] - hf://facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: TOKIO_WORKER_THREADS [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: HF_HUB_ENABLE_HF_TRANSFER [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_HIGH_PERFORMANCE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_NUM_CONCURRENT_RANGE_GETS [e2e-llm-inference-service] value: '8' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n # Check if\ [e2e-llm-inference-service] \ we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ # Join the array elements with a comma\n hca_port_pairs=()\n \ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n hca_port_pairs+=(\"\ [e2e-llm-inference-service] ${hca}:1\")\n done\n\n active_hca_list=$(IFS=,; echo \"${active_hcas[*]}\"\ [e2e-llm-inference-service] )\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\n# --kv-transfer-config with OffloadingConnector requires vLLM\ [e2e-llm-inference-service] \ 0.22.0+ (vllm-project/vllm#40020).\nKV_TRANSFER_ARGS=\"\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.22.0\" ]; then\n if [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\" !=\ [e2e-llm-inference-service] \ *\"--kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"\ [e2e-llm-inference-service] * ]] && [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n KV_TRANSFER_ARGS=\"\ [e2e-llm-inference-service] \"\n fi\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \"publishers/e2e-test-llm-inference-service-fac1c4ae/models/facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8000 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n ${KV_TRANSFER_ARGS} \\\n \\\n --enable-ssl-refresh \\\n --ssl-certfile\ [e2e-llm-inference-service] \ /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-s8czmr [e2e-llm-inference-service] generateName: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-fac1c4ae [e2e-llm-inference-service] uid: 5ed49ea7-fc7d-4240-ad6d-65b3dcee042f [e2e-llm-inference-service] resourceVersion: '22366' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:29:43Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-3c960099 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-28T17:29:43Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] uid: ce01253e-1e45-4bb3-af6d-6d10bb9b4ffb [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T17:29:44Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"ce01253e-1e45-4bb3-af6d-6d10bb9b4ffb"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.134.0.25 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: false [e2e-llm-inference-service] serving: false [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-fac1c4ae [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-788d7df956lftbb [e2e-llm-inference-service] uid: ec8c770f-4829-4d70-809d-2d31b8297a9a [e2e-llm-inference-service] nodeName: ip-10-0-135-20.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:29:43Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-3c960099 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"5c6f11ac-adba-4e63-8af1-189e8f5b3f44"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T17:29:43Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-28T17:29:43Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-28T17:29:44Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-route [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-fac1c4ae [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099 [e2e-llm-inference-service] uid: 5c6f11ac-adba-4e63-8af1-189e8f5b3f44 [e2e-llm-inference-service] resourceVersion: '22424' [e2e-llm-inference-service] uid: eda12794-93d9-4c5b-8a52-4f92ae2364d7 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-fac1c4ae/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-fac1c4ae/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-fac1c4ae/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-fac1c4ae/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-fac1c4ae/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-fac1c4ae/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-fac1c4ae/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-fac1c4ae/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-fac1c4ae/models/facebook/opt-125m/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-fac1c4ae/models/facebook/opt-125m/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-fac1c4ae/models/facebook/opt-125m/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-fac1c4ae/models/facebook/opt-125m/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-fac1c4ae/models/facebook/opt-125m [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099 [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-fac1c4ae/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T17:29:43Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T17:29:43Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T17:29:44Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:29:43Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-3c960099 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"5c6f11ac-adba-4e63-8af1-189e8f5b3f44"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T17:29:43Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-28T17:29:43Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-28T17:29:44Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-route [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-fac1c4ae [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099 [e2e-llm-inference-service] uid: 5c6f11ac-adba-4e63-8af1-189e8f5b3f44 [e2e-llm-inference-service] resourceVersion: '22424' [e2e-llm-inference-service] uid: eda12794-93d9-4c5b-8a52-4f92ae2364d7 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-fac1c4ae/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-fac1c4ae/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-fac1c4ae/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-fac1c4ae/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-fac1c4ae/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-fac1c4ae/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-fac1c4ae/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-fac1c4ae/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-fac1c4ae/models/facebook/opt-125m/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-fac1c4ae/models/facebook/opt-125m/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-fac1c4ae/models/facebook/opt-125m/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-fac1c4ae/models/facebook/opt-125m/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-fac1c4ae/models/facebook/opt-125m [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099 [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-fac1c4ae/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T17:29:43Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T17:29:43Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T17:29:44Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] kind: AuthPolicy [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:29:44Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-policies [e2e-llm-inference-service] app.kubernetes.io/managed-by: odh-model-controller [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-3c960099 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"5c6f11ac-adba-4e63-8af1-189e8f5b3f44"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:rules: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:authentication: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:public: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:anonymous: {} [e2e-llm-inference-service] f:credentials: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:overrides: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fairness: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:response: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:success: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:headers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:x-gateway-inference-fairness-id: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:x-gateway-inference-objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:targetRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T17:29:44Z' [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Accepted"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Enforced"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-28T17:29:46Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-route-authn [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-fac1c4ae [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099 [e2e-llm-inference-service] uid: 5c6f11ac-adba-4e63-8af1-189e8f5b3f44 [e2e-llm-inference-service] resourceVersion: '22448' [e2e-llm-inference-service] uid: 488d17d0-b101-49d3-82a6-f640362b38af [e2e-llm-inference-service] spec: [e2e-llm-inference-service] rules: [e2e-llm-inference-service] authentication: [e2e-llm-inference-service] public: [e2e-llm-inference-service] anonymous: {} [e2e-llm-inference-service] credentials: {} [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] overrides: [e2e-llm-inference-service] fairness: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] objective: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] response: [e2e-llm-inference-service] success: [e2e-llm-inference-service] headers: [e2e-llm-inference-service] x-gateway-inference-fairness-id: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.fairness [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] x-gateway-inference-objective: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.objective [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-route [e2e-llm-inference-service] status: [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T17:29:44Z' [e2e-llm-inference-service] message: AuthPolicy has been accepted [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T17:29:46Z' [e2e-llm-inference-service] message: AuthPolicy has been successfully enforced [e2e-llm-inference-service] reason: Enforced [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Enforced [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:29:43Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-3c960099 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"5c6f11ac-adba-4e63-8af1-189e8f5b3f44"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T17:29:43Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-fac1c4ae [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099 [e2e-llm-inference-service] uid: 5c6f11ac-adba-4e63-8af1-189e8f5b3f44 [e2e-llm-inference-service] resourceVersion: '22306' [e2e-llm-inference-service] uid: f941a624-8418-4aad-a744-5b4f5103de05 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc.e2e-test-llm-inference-service-fac1c4ae.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc.e2e-test-llm-inference-service-fac1c4ae.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:29:43Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-3c960099 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"5c6f11ac-adba-4e63-8af1-189e8f5b3f44"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T17:29:43Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-fac1c4ae [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099 [e2e-llm-inference-service] uid: 5c6f11ac-adba-4e63-8af1-189e8f5b3f44 [e2e-llm-inference-service] resourceVersion: '22306' [e2e-llm-inference-service] uid: f941a624-8418-4aad-a744-5b4f5103de05 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc.e2e-test-llm-inference-service-fac1c4ae.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc.e2e-test-llm-inference-service-fac1c4ae.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:29:43Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-3c960099 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"5c6f11ac-adba-4e63-8af1-189e8f5b3f44"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T17:29:43Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-fac1c4ae [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099 [e2e-llm-inference-service] uid: 5c6f11ac-adba-4e63-8af1-189e8f5b3f44 [e2e-llm-inference-service] resourceVersion: '22306' [e2e-llm-inference-service] uid: f941a624-8418-4aad-a744-5b4f5103de05 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc.e2e-test-llm-inference-service-fac1c4ae.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc.e2e-test-llm-inference-service-fac1c4ae.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_inference_service] [2026-07-28T17:44:13.566331] end - ❌ 902.173s: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T17:29:44Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T17:30:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:29:44Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T17:30:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T17:29:44Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T17:30:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ---------------------------- Captured log teardown ----------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.conftest:conftest.py:168 Skipping deletion of namespace e2e-test-llm-inference-service-fac1c4ae (SKIP_DELETION_ON_FAILURE) [e2e-llm-inference-service] __________ test_llm_auth_enabled_requires_token[auth-enabled-default] __________ [e2e-llm-inference-service] [gw0] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-single-cpu', 'model-fb-opt-125m'], prompt='KServe is a', service_name=... {'name': 'model-fb-opt-125m-auth-enabled-89f54b63'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] @pytest.mark.auth [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="auth-enabled-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] ], [e2e-llm-inference-service] id="auth-enabled-default", [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_auth_enabled_requires_token(test_case: TestCase): # noqa: F811 [e2e-llm-inference-service] """ [e2e-llm-inference-service] Test that when auth is enabled (default): [e2e-llm-inference-service] - Requests WITH valid token succeed [e2e-llm-inference-service] - Requests WITHOUT token are rejected (401/403) [e2e-llm-inference-service] """ [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = KServeClient( [e2e-llm-inference-service] config_file=os.environ.get("KUBECONFIG", "~/.kube/config"), [e2e-llm-inference-service] client_configuration=client.Configuration(), [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] sa_name = f"{service_name}-test-sa" [e2e-llm-inference-service] ns = test_case.llm_service.metadata.namespace [e2e-llm-inference-service] test_failed = False [e2e-llm-inference-service] [e2e-llm-inference-service] # Enable auth for this test [e2e-llm-inference-service] if not test_case.llm_service.metadata.annotations: [e2e-llm-inference-service] test_case.llm_service.metadata.annotations = {} [e2e-llm-inference-service] test_case.llm_service.metadata.annotations[ [e2e-llm-inference-service] "security.opendatahub.io/enable-auth" [e2e-llm-inference-service] ] = "true" [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # Create LLMInferenceService [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_auth.py:276: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...enable-a18fd8e2'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-auth-enabled-89f54b63'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-07-28T17:32:15.849237', start_time = 1785259935.8494997 [e2e-llm-inference-service] duration = 900.8128845691681, timestamp_end = '2026-07-28T17:47:16.662387' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....-auth-enable-a18fd8e2'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-auth-enabled-89f54b63'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1376: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7f67b9cd47c0> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1387: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T17:32:38Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T17:32:38Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T17:32:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:32:26Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T17:32:38Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T17:32:57Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T17:32:57Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:32:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1371: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:82 Created test namespace e2e-test-llm-auth-enabled-requires-token-d4ed2c08 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret seaweedfs-s3-creds from kserve-ci-e2e-test to e2e-test-llm-auth-enabled-requires-token-d4ed2c08 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret storage-config from kserve-ci-e2e-test to e2e-test-llm-auth-enabled-requires-token-d4ed2c08 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:178 Patched default SA in e2e-test-llm-auth-enabled-requires-token-d4ed2c08 with secret seaweedfs-s3-creds [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:162 ConfigMap odh-kserve-custom-ca-bundle already exists in e2e-test-llm-auth-enabled-requires-token-d4ed2c08 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:159 Copied ConfigMap odh-trusted-ca-bundle from kserve-ci-e2e-test to e2e-test-llm-auth-enabled-requires-token-d4ed2c08 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig router-managed-auth-enabled-tes-210a8f79 in namespace e2e-test-llm-auth-enabled-requires-token-d4ed2c08 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig router-managed-auth-enabled-tes-210a8f79 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig router-managed-auth-enabled-tes-210a8f79 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig workload-single-cpu-auth-enable-a18fd8e2 in namespace e2e-test-llm-auth-enabled-requires-token-d4ed2c08 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig workload-single-cpu-auth-enable-a18fd8e2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig workload-single-cpu-auth-enable-a18fd8e2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig model-fb-opt-125m-auth-enabled-89f54b63 in namespace e2e-test-llm-auth-enabled-requires-token-d4ed2c08 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig model-fb-opt-125m-auth-enabled-89f54b63 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig model-fb-opt-125m-auth-enabled-89f54b63 [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_auth_enabled_requires_token] [2026-07-28T17:32:15.693119] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-single-cpu', 'model-fb-opt-125m'], prompt='KServe is a', service_name='auth-enabled-test', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, namespace='e2e-test-llm-auth-enabled-requires-token-d4ed2c08', before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'auth-enabled-test', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-auth-enabled-requires-token-d4ed2c08', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-auth-enabled-tes-210a8f79'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-auth-enable-a18fd8e2'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-auth-enabled-89f54b63'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-28T17:32:15.713036] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'true'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'auth-enabled-test', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-auth-enabled-requires-token-d4ed2c08', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-auth-enabled-tes-210a8f79'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-auth-enable-a18fd8e2'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-auth-enabled-89f54b63'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-28T17:32:15.849067] end - ✅ in 0.136s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-28T17:32:15.849237] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'true'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'auth-enabled-test', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-auth-enabled-requires-token-d4ed2c08', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-auth-enabled-tes-210a8f79'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-auth-enable-a18fd8e2'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-auth-enabled-89f54b63'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'RouterReady', 'WorkloadsReady'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T17:32:26Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T17:32:26Z', 'message': 'Inference Pool e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T17:32:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:32:26Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T17:32:26Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T17:32:26Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T17:32:26Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:32:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'RouterReady', 'WorkloadsReady'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T17:32:38Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T17:32:38Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T17:32:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:32:26Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T17:32:38Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T17:32:38Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T17:32:38Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:32:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T17:32:38Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T17:32:38Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T17:32:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:32:26Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T17:32:38Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T17:32:57Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T17:32:57Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:32:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 1: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-28T17:29:27Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 2: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-28T17:29:27Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 3: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-28T17:29:27Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1391 Timed out waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T17:32:38Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T17:32:38Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T17:32:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:32:26Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T17:32:38Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T17:32:57Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T17:32:57Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:32:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-07-28T17:47:16.662387] end - ❌ 900.813s: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T17:32:38Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T17:32:38Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T17:32:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:32:26Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T17:32:38Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T17:32:57Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T17:32:57Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:32:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_auth.py:346 ❌ ERROR: Failed test for auth-enabled-test: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T17:32:38Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T17:32:38Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T17:32:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:32:26Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T17:32:38Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T17:32:57Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T17:32:57Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:32:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:243 # Diagnostics for 'auth-enabled-test' in 'e2e-test-llm-auth-enabled-requires-token-d4ed2c08' [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:244 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:247 # LLMInferenceService auth-enabled-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:256 apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] security.opendatahub.io/enable-auth: 'true' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:32:15Z' [e2e-llm-inference-service] finalizers: [e2e-llm-inference-service] - serving.kserve.io/llmisvc-finalizer [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:security.opendatahub.io/enable-auth: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:baseRefs: {} [e2e-llm-inference-service] manager: OpenAPI-Generator [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T17:32:15Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:finalizers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] v:"serving.kserve.io/llmisvc-finalizer": {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T17:32:15Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:addresses: {} [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-router-route: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-scheduler: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-scheduler-latency-predictor: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-tokenizer: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-tracing: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-worker-data-parallel: {} [e2e-llm-inference-service] f:appliedConfigs: {} [e2e-llm-inference-service] f:conditions: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:router: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:gateways: {} [e2e-llm-inference-service] f:scheduler: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:inferencePool: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:service: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:url: {} [e2e-llm-inference-service] f:workloads: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:primary: {} [e2e-llm-inference-service] f:scheduler: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-28T17:32:57Z' [e2e-llm-inference-service] name: auth-enabled-test [e2e-llm-inference-service] namespace: e2e-test-llm-auth-enabled-requires-token-d4ed2c08 [e2e-llm-inference-service] resourceVersion: '26140' [e2e-llm-inference-service] uid: 3f4fbd10-5888-4617-9e6f-f440212e4c59 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] baseRefs: [e2e-llm-inference-service] - name: router-managed-auth-enabled-tes-210a8f79 [e2e-llm-inference-service] - name: workload-single-cpu-auth-enable-a18fd8e2 [e2e-llm-inference-service] - name: model-fb-opt-125m-auth-enabled-89f54b63 [e2e-llm-inference-service] model: [e2e-llm-inference-service] name: auth-enabled-test [e2e-llm-inference-service] uri: '' [e2e-llm-inference-service] status: [e2e-llm-inference-service] addresses: [e2e-llm-inference-service] - name: gateway-external-model-routing [e2e-llm-inference-service] url: http://a803f1ae9e25d4fc48c1c4d8072319d1-1138438229.us-east-1.elb.amazonaws.com/ [e2e-llm-inference-service] - name: gateway-external [e2e-llm-inference-service] url: http://a803f1ae9e25d4fc48c1c4d8072319d1-1138438229.us-east-1.elb.amazonaws.com/e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test [e2e-llm-inference-service] - name: gateway-external [e2e-llm-inference-service] url: http://a803f1ae9e25d4fc48c1c4d8072319d1-1138438229.us-east-1.elb.amazonaws.com/publishers/e2e-test-llm-auth-enabled-requires-token-d4ed2c08/models/facebook/opt-125m [e2e-llm-inference-service] - name: gateway-internal-model-routing [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/ [e2e-llm-inference-service] - name: gateway-internal [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test [e2e-llm-inference-service] - name: gateway-internal [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/publishers/e2e-test-llm-auth-enabled-requires-token-d4ed2c08/models/facebook/opt-125m [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-template: kserve-config-llm-decode-template [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-worker-data-parallel: kserve-config-llm-decode-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-template: kserve-config-llm-prefill-template [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-worker-data-parallel: kserve-config-llm-prefill-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-router-route: kserve-config-llm-router-route [e2e-llm-inference-service] serving.kserve.io/config-llm-scheduler: kserve-config-llm-scheduler [e2e-llm-inference-service] serving.kserve.io/config-llm-scheduler-latency-predictor: kserve-config-llm-scheduler-latency-predictor [e2e-llm-inference-service] serving.kserve.io/config-llm-template: kserve-config-llm-template [e2e-llm-inference-service] serving.kserve.io/config-llm-tokenizer: kserve-config-llm-tokenizer [e2e-llm-inference-service] serving.kserve.io/config-llm-tracing: kserve-config-llm-tracing [e2e-llm-inference-service] serving.kserve.io/config-llm-worker-data-parallel: kserve-config-llm-worker-data-parallel [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T17:32:38Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: HTTPRoutesReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T17:32:38Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: InferencePoolReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T17:32:26Z' [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: MainWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T17:32:26Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: PresetsCombined [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T17:32:38Z' [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: Ready [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T17:32:57Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: RouterReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T17:32:57Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: SchedulerWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T17:32:26Z' [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: WorkloadsReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] url: http://a803f1ae9e25d4fc48c1c4d8072319d1-1138438229.us-east-1.elb.amazonaws.com/e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:68 TIME NAMESPACE SOURCE TYPE REASON MESSAGE [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:69 -------------------------------------------------------------------------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None e2e-test-llm-auth-enabled-requires-token-d4ed2c08 Normal Scheduled Successfully assigned e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-766449dfcf-hn46v to ip-10-0-140-52.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 17:32:25 e2e-test-llm-auth-enabled-requires-token-d4ed2c08 multus Normal AddedInterface Add eth0 [10.133.0.46/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 17:32:25 e2e-test-llm-auth-enabled-requires-token-d4ed2c08 kubelet/ip-10-0-140-52.ec2.internal Normal Pulling Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 17:32:28 e2e-test-llm-auth-enabled-requires-token-d4ed2c08 kubelet/ip-10-0-140-52.ec2.internal Normal Pulled Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" in 3.814s (3.814s including waiting). Image size: 300915674 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 17:32:28 e2e-test-llm-auth-enabled-requires-token-d4ed2c08 kubelet/ip-10-0-140-52.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 17:32:28 e2e-test-llm-auth-enabled-requires-token-d4ed2c08 kubelet/ip-10-0-140-52.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 17:32:24 e2e-test-llm-auth-enabled-requires-token-d4ed2c08 replicaset-controller Normal SuccessfulCreate Created pod: auth-enabled-test-kserve-766449dfcf-hn46v [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None e2e-test-llm-auth-enabled-requires-token-d4ed2c08 Normal Scheduled Successfully assigned e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-hgkl8 to ip-10-0-135-188.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 17:32:25 e2e-test-llm-auth-enabled-requires-token-d4ed2c08 multus Normal AddedInterface Add eth0 [10.132.0.37/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 17:32:25 e2e-test-llm-auth-enabled-requires-token-d4ed2c08 kubelet/ip-10-0-135-188.ec2.internal Normal Pulled Container image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 17:32:25 e2e-test-llm-auth-enabled-requires-token-d4ed2c08 kubelet/ip-10-0-135-188.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 17:32:25 e2e-test-llm-auth-enabled-requires-token-d4ed2c08 kubelet/ip-10-0-135-188.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 17:32:24 e2e-test-llm-auth-enabled-requires-token-d4ed2c08 replicaset-controller Normal SuccessfulCreate Created pod: auth-enabled-test-kserve-router-scheduler-5dbb5856d8-hgkl8 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 17:32:24 e2e-test-llm-auth-enabled-requires-token-d4ed2c08 deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-enabled-test-kserve-router-scheduler-5dbb5856d8 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 17:32:24 e2e-test-llm-auth-enabled-requires-token-d4ed2c08 deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-enabled-test-kserve-766449dfcf from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 17:32:24 e2e-test-llm-auth-enabled-requires-token-d4ed2c08 LLMInferenceServiceController Normal Created Created v1.Secret e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 17:32:24 e2e-test-llm-auth-enabled-requires-token-d4ed2c08 LLMInferenceServiceController Normal Created Created v1.Deployment e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 17:32:24 e2e-test-llm-auth-enabled-requires-token-d4ed2c08 LLMInferenceServiceController Normal Created Created v1.Service e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 17:32:24 e2e-test-llm-auth-enabled-requires-token-d4ed2c08 LLMInferenceServiceController Normal Created Created v1.ServiceAccount e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 17:32:24 e2e-test-llm-auth-enabled-requires-token-d4ed2c08 LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /e6eba5ecf557d59a5eb263733b3690452-auth-enabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 17:32:24 e2e-test-llm-auth-enabled-requires-token-d4ed2c08 LLMInferenceServiceController Normal Created Created v1.Role e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 17:32:24 e2e-test-llm-auth-enabled-requires-token-d4ed2c08 LLMInferenceServiceController Normal Created Created v1.RoleBinding e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 17:32:24 e2e-test-llm-auth-enabled-requires-token-d4ed2c08 LLMInferenceServiceController Normal Created Created v1.Deployment e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 17:32:25 e2e-test-llm-auth-enabled-requires-token-d4ed2c08 LLMInferenceServiceController Normal Created Created v1.Service e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 17:32:38 e2e-test-llm-auth-enabled-requires-token-d4ed2c08 LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 17:32:49 e2e-test-llm-auth-enabled-requires-token-d4ed2c08 LLMInferenceServiceController Normal Updated Updated v1.Secret e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 17:32:37 e2e-test-llm-auth-enabled-requires-token-d4ed2c08 LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:172 ### Pod auth-enabled-test-kserve-766449dfcf-hn46v (phase=Pending) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:212 #### init-container 'storage-initializer' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:225 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:226 2026-07-28 17:32:29.203 1 storage.initializer INFO [initializer-entrypoint:():17] Initializing, args: (src_uri, dest_path): [('hf://facebook/opt-125m', '/mnt/models')] [e2e-llm-inference-service] 2026-07-28 17:32:29.203 1 storage.initializer INFO [kserve_storage.py:download():168] Copying contents of hf://facebook/opt-125m to local [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/wPaCkH-WbT7GsmxMKKrNZTV4nSM=.ac481c8eb05e4d2496fbe076a38a7b4835dd733d.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_b6b9f16e-78a0-4df5-ad7e-e545592a0494'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/5HHJ6px3_ZRDOG3OxNZMhuycwOk=.a591333512516f58bf2002045dece909a0ccdb8b.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_62eb6925-3dd1-468f-95b7-5549637c7110'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/Xn7B-BWUGOee2Y6hCZtEhtFu4BE=.38c05904caf6e5b9f04ecda5c973d77e6c1da151.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_ebb86076-2028-4574-b95d-c32f1d4a666e'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/8_PA_wEVGiVa2goH2H4KQOQpvVY=.b3fb716a3024261980becb2382e31a3780985130.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_5a079cbe-7792-4247-b26c-f3fe9f00a2a9'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:212 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:228 # -- logs (current): unavailable ((400) [e2e-llm-inference-service] Reason: Bad Request [e2e-llm-inference-service] HTTP response headers: HTTPHeaderDict({'Audit-Id': 'ef6bb293-a196-4f02-8f4b-49bf104e2e2d', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'Date': 'Tue, 28 Jul 2026 17:47:16 GMT', 'Content-Length': '223'}) [e2e-llm-inference-service] HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"container \"main\" in pod \"auth-enabled-test-kserve-766449dfcf-hn46v\" is waiting to start: PodInitializing","reason":"BadRequest","code":400} [e2e-llm-inference-service] [e2e-llm-inference-service] ) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:172 ### Pod auth-enabled-test-kserve-router-scheduler-5dbb5856d8-hgkl8 (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:212 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:225 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:226 {"level":"info","ts":1785259945.5309741,"logger":"setup","caller":"runner/runner.go:196","msg":"GIE build","commit-sha":"unknown","build-ref":""} [e2e-llm-inference-service] {"level":"info","ts":1785259945.5310946,"logger":"setup","caller":"runner/runner.go:217","msg":"Flags processed","flags":{"cert-path":"/var/run/kserve/tls","config-file":"","config-text":"apiVersion: llm-d.ai/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n- type: single-profile-handler\n- type: queue-scorer\n- type: kv-cache-utilization-scorer\n- type: prefix-cache-scorer\n- type: no-hit-lru-scorer\n- type: max-score-picker\n- parameters:\n scheme: https\n type: metrics-data-source\nschedulingProfiles:\n- name: default\n plugins:\n - pluginRef: queue-scorer\n weight: 2\n - pluginRef: kv-cache-utilization-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: no-hit-lru-scorer\n weight: 2\n - pluginRef: max-score-picker\n","disable-endpoint-subset-filter":false,"enable-cert-reload":true,"enable-grpc-stream-metrics":false,"enable-pprof":true,"endpoint-selector":"","endpoint-target-ports":{},"grpc-health-port":9003,"grpc-max-recv-msg-size":"","grpc-max-send-msg-size":"","grpc-port":9002,"ha-enable-leader-election":false,"health-checking":false,"metrics-endpoint-auth":true,"metrics-port":9090,"metrics-staleness-threshold":2000000000,"model-server-metrics-https-insecure-skip-verify":true,"model-server-metrics-path":"/metrics","model-server-metrics-port":0,"model-server-metrics-scheme":"http","pool-group":"inference.networking.k8s.io","pool-name":"auth-enabled-test-inference-pool","pool-namespace":"e2e-test-llm-auth-enabled-requires-token-d4ed2c08","refresh-metrics-interval":50000000,"refresh-prometheus-metrics-interval":5000000000,"secure-serving":true,"tracing":true,"v":2,"zap-devel":{},"zap-encoder":{},"zap-log-level":{},"zap-stacktrace-level":{},"zap-time-encoding":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785259945.5312138,"logger":"setup.trace","caller":"tracing/telemetry.go:123","msg":"init OTel trace exporter","type":"console"} [e2e-llm-inference-service] {"level":"info","ts":1785259945.5317183,"caller":"loader/configloader.go:121","msg":"Loaded raw configuration","config":"{Plugins: [{Type: single-profile-handler} {Type: queue-scorer} {Type: kv-cache-utilization-scorer} {Type: prefix-cache-scorer} {Type: no-hit-lru-scorer} {Type: max-score-picker} {Type: metrics-data-source, Parameters: {\"scheme\":\"https\"}}], SchedulingProfiles: [{Name: default, Plugins: [{PluginRef: queue-scorer, Weight: 2.00} {PluginRef: kv-cache-utilization-scorer, Weight: 2.00} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: no-hit-lru-scorer, Weight: 2.00} {PluginRef: max-score-picker}]}]}"} [e2e-llm-inference-service] {"level":"info","ts":1785259945.531745,"logger":"setup","caller":"runner/runner.go:622","msg":"Data layer: ENABLED"} [e2e-llm-inference-service] {"level":"info","ts":1785259945.5319905,"logger":"setup","caller":"runner/runner.go:281","msg":"Raw config after phase one","config":{"apiVersion":"llm-d.ai/v1alpha1","dataLayer":null,"kind":"EndpointPickerConfig","plugins":[{"name":"single-profile-handler","parameters":null,"type":"single-profile-handler"},{"name":"queue-scorer","parameters":null,"type":"queue-scorer"},{"name":"kv-cache-utilization-scorer","parameters":null,"type":"kv-cache-utilization-scorer"},{"name":"prefix-cache-scorer","parameters":null,"type":"prefix-cache-scorer"},{"name":"no-hit-lru-scorer","parameters":null,"type":"no-hit-lru-scorer"},{"name":"max-score-picker","parameters":null,"type":"max-score-picker"},{"name":"metrics-data-source","parameters":{"scheme":"https"},"type":"metrics-data-source"}],"schedulingProfiles":[{"name":"default","plugins":[{"pluginRef":"queue-scorer","weight":2},{"pluginRef":"kv-cache-utilization-scorer","weight":2},{"pluginRef":"prefix-cache-scorer","weight":3},{"pluginRef":"no-hit-lru-scorer","weight":2},{"pluginRef":"max-score-picker","weight":null}]}]}} [e2e-llm-inference-service] {"level":"info","ts":1785259945.5504866,"logger":"utilization-detector/utilization-detector","caller":"utilization/detector.go:83","msg":"Creating new UtilizationDetector","queueDepthThreshold":5,"kvCacheUtilThreshold":0.8,"metricsStalenessThreshold":"200ms","headroom":0} [e2e-llm-inference-service] {"level":"info","ts":1785259945.550595,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"vllm","mapping":"Mapping{all specs enabled}"} [e2e-llm-inference-service] {"level":"info","ts":1785259945.5506308,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"sglang","mapping":"Mapping{disabled: [lora]}"} [e2e-llm-inference-service] {"level":"info","ts":1785259945.5506673,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"trtllm-serve","mapping":"Mapping{disabled: [lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1785259945.5507267,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"triton-tensorrt-llm","mapping":"Mapping{disabled: [lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1785259945.550746,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"triton","mapping":"Mapping{disabled: [kv, lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1785259945.5507905,"caller":"loader/configloader.go:154","msg":"Instantiated all plugins and applied system defaults. Effective raw configuration","config":"{Plugins: [{Name: single-profile-handler, Type: single-profile-handler} {Name: queue-scorer, Type: queue-scorer} {Name: kv-cache-utilization-scorer, Type: kv-cache-utilization-scorer} {Name: prefix-cache-scorer, Type: prefix-cache-scorer} {Name: no-hit-lru-scorer, Type: no-hit-lru-scorer} {Name: max-score-picker, Type: max-score-picker} {Name: metrics-data-source, Type: metrics-data-source, Parameters: {\"scheme\":\"https\"}} {Name: fcfs-ordering-policy, Type: fcfs-ordering-policy} {Name: global-strict-fairness-policy, Type: global-strict-fairness-policy} {Name: static-usage-limit-policy, Type: static-usage-limit-policy} {Name: openai-parser, Type: openai-parser} {Name: anthropic-parser, Type: anthropic-parser} {Name: vllmhttp-parser, Type: vllmhttp-parser} {Name: utilization-detector, Type: utilization-detector} {Name: core-metrics-extractor, Type: core-metrics-extractor}], SchedulingProfiles: [{Name: default, Plugins: [{PluginRef: queue-scorer, Weight: 2.00} {PluginRef: kv-cache-utilization-scorer, Weight: 2.00} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: no-hit-lru-scorer, Weight: 2.00} {PluginRef: max-score-picker}]}], DataLayer: {Sources: [{PluginRef: metrics-data-source, Extractors: [{PluginRef: core-metrics-extractor}]}], Discovery: }, FlowControl: {MaxBytes: unlimited, MaxRequests: unlimited, SaturationDetector: {PluginRef: utilization-detector}}, RequestHandler: {Parsers: [{PluginRef: openai-parser}, {PluginRef: anthropic-parser}, {PluginRef: vllmhttp-parser}]}}"} [e2e-llm-inference-service] {"level":"info","ts":1785259945.5508733,"caller":"approximateprefix/plugin.go:88","msg":"Prefix DataProducer initialized","config":{"autoTune":true,"blockSizeTokens":16,"blockSize":0,"maxPrefixBlocksToMatch":2048,"maxPrefixTokensToMatch":131072,"lruCapacityPerServer":31250}} [e2e-llm-inference-service] {"level":"info","ts":1785259945.5509326,"caller":"approximateprefix/plugin.go:111","msg":"WARNING: configured blockSizeTokens is below the recommended minimum, overriding it.","blockSizeTokens":16,"minimum":64,"issue":"https://github.com/llm-d/llm-d-router/issues/1158"} [e2e-llm-inference-service] {"level":"info","ts":1785259945.550951,"caller":"datalayer/data_graph.go:116","msg":"auto-created default producer","producer":"approx-prefix-cache-producer/approx-prefix-cache-producer","dataKey":"PrefixCacheMatchInfoDataKey/approx-prefix-cache-producer","consumer":"prefix-cache-scorer"} [e2e-llm-inference-service] {"level":"info","ts":1785259945.5509732,"caller":"datalayer/data_graph.go:116","msg":"auto-created default producer","producer":"token-producer/token-producer","dataKey":"TokenizedPrompt/token-producer","consumer":"approx-prefix-cache-producer"} [e2e-llm-inference-service] {"level":"info","ts":1785259945.5510728,"caller":"runner/runner.go:685","msg":"loaded configuration from file/text successfully"} [e2e-llm-inference-service] {"level":"info","ts":1785259945.5510833,"logger":"setup","caller":"runner/runner.go:308","msg":"EPP config after phase two","config":"{SchedulerConfig:{ProfileHandler: single-profile-handler/single-profile-handler, Profiles: map[default:{Filters: [], Scorers: [queue-scorer/queue-scorer: 2.000000, kv-cache-utilization-scorer/kv-cache-utilization-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000, no-hit-lru-scorer/no-hit-lru-scorer: 2.000000], Picker: max-score-picker/max-score-picker}]} SaturationDetector:0xc000978240 DataConfig:{Sources:[{Plugin:0xc000a0e240 Extractors:[0xc000978440]}]} FlowControlConfig: ParserRegistry:0xc000978980}"} [e2e-llm-inference-service] {"level":"info","ts":1785259945.5682294,"logger":"setup","caller":"runner/runner.go:352","msg":"Setting pprof handlers"} [e2e-llm-inference-service] {"level":"info","ts":1785259945.56826,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/block"} [e2e-llm-inference-service] {"level":"info","ts":1785259945.568274,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/cmdline"} [e2e-llm-inference-service] {"level":"info","ts":1785259945.5682797,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/profile"} [e2e-llm-inference-service] {"level":"info","ts":1785259945.5682845,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/symbol"} [e2e-llm-inference-service] {"level":"info","ts":1785259945.5682893,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/heap"} [e2e-llm-inference-service] {"level":"info","ts":1785259945.568293,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/allocs"} [e2e-llm-inference-service] {"level":"info","ts":1785259945.5682976,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/mutex"} [e2e-llm-inference-service] {"level":"info","ts":1785259945.568302,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/"} [e2e-llm-inference-service] {"level":"info","ts":1785259945.5683064,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/trace"} [e2e-llm-inference-service] {"level":"info","ts":1785259945.5683107,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/goroutine"} [e2e-llm-inference-service] {"level":"info","ts":1785259945.5683148,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/threadcreate"} [e2e-llm-inference-service] {"level":"info","ts":1785259945.5683217,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/plugins/state"} [e2e-llm-inference-service] {"level":"info","ts":1785259945.5683265,"logger":"setup","caller":"runner/runner.go:373","msg":"parsed config","scheduler-config":"{ProfileHandler: single-profile-handler/single-profile-handler, Profiles: map[default:{Filters: [], Scorers: [queue-scorer/queue-scorer: 2.000000, kv-cache-utilization-scorer/kv-cache-utilization-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000, no-hit-lru-scorer/no-hit-lru-scorer: 2.000000], Picker: max-score-picker/max-score-picker}]}"} [e2e-llm-inference-service] {"level":"info","ts":1785259945.5683522,"logger":"setup","caller":"datalayer/runtime.go:99","msg":"Configuring datalayer runtime","numSources":1} [e2e-llm-inference-service] {"level":"info","ts":1785259945.56836,"logger":"setup","caller":"datalayer/runtime.go:118","msg":"Processing source","source":"metrics-data-source","numExtractors":1} [e2e-llm-inference-service] {"level":"info","ts":1785259945.5683732,"logger":"setup","caller":"datalayer/runtime.go:147","msg":"Source configured","source":"metrics-data-source","extractors":["core-metrics-extractor/core-metrics-extractor"]} [e2e-llm-inference-service] {"level":"info","ts":1785259945.568385,"logger":"setup","caller":"datalayer/runtime.go:206","msg":"Datalayer runtime configured","pollers":1,"notifiers":0,"endpointSources":0} [e2e-llm-inference-service] {"level":"info","ts":1785259945.5683935,"logger":"setup","caller":"runner/runner.go:833","msg":"Experimental Flow Control layer is disabled, using legacy admission control"} [e2e-llm-inference-service] {"level":"info","ts":1785259945.5684838,"logger":"setup","caller":"runner/runner.go:721","msg":"ExtProc server runner added to manager."} [e2e-llm-inference-service] {"level":"info","ts":1785259945.5684934,"logger":"setup","caller":"runner/runner.go:260","msg":"Controller manager starting"} [e2e-llm-inference-service] {"level":"info","ts":1785259945.5685325,"logger":"controller-runtime.metrics","caller":"server/server.go:208","msg":"Starting metrics server"} [e2e-llm-inference-service] {"level":"info","ts":1785259945.5687373,"caller":"runnable/grpc.go:35","msg":"gRPC server starting","name":"health"} [e2e-llm-inference-service] {"level":"info","ts":1785259945.568931,"caller":"runnable/grpc.go:43","msg":"gRPC server listening","name":"health","port":9003} [e2e-llm-inference-service] {"level":"info","ts":1785259945.5691056,"logger":"controller-runtime.metrics","caller":"server/server.go:247","msg":"Serving metrics server","bindAddress":":9090","secure":false} [e2e-llm-inference-service] {"level":"info","ts":1785259945.569115,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferenceobjective","controllerGroup":"llm-d.ai","controllerKind":"InferenceObjective","source":"kind source: *v1alpha2.InferenceObjective"} [e2e-llm-inference-service] {"level":"info","ts":1785259945.5691633,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","source":"kind source: *v1.InferencePool"} [e2e-llm-inference-service] {"level":"info","ts":1785259945.5691996,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"pod","controllerGroup":"","controllerKind":"Pod","source":"kind source: *v1.Pod"} [e2e-llm-inference-service] {"level":"info","ts":1785259945.5692828,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferencemodelrewrite","controllerGroup":"llm-d.ai","controllerKind":"InferenceModelRewrite","source":"kind source: *v1alpha2.InferenceModelRewrite"} [e2e-llm-inference-service] {"level":"info","ts":1785259945.5698476,"caller":"runnable/grpc.go:35","msg":"gRPC server starting","name":"ext-proc"} [e2e-llm-inference-service] {"level":"info","ts":1785259945.5699537,"caller":"runnable/grpc.go:43","msg":"gRPC server listening","name":"ext-proc","port":9002} [e2e-llm-inference-service] {"level":"info","ts":1785259945.5735767,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.InferenceObjective","reflector":"go/pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1785259945.573582,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.InferenceModelRewrite","reflector":"go/pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1785259945.5737672,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.InferencePool","reflector":"go/pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1785259945.5743117,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Pod","reflector":"go/pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1785259945.6701882,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferencemodelrewrite","controllerGroup":"llm-d.ai","controllerKind":"InferenceModelRewrite"} [e2e-llm-inference-service] {"level":"info","ts":1785259945.6702409,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferencemodelrewrite","controllerGroup":"llm-d.ai","controllerKind":"InferenceModelRewrite","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1785259945.6702127,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"pod","controllerGroup":"","controllerKind":"Pod"} [e2e-llm-inference-service] {"level":"info","ts":1785259945.6702044,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferenceobjective","controllerGroup":"llm-d.ai","controllerKind":"InferenceObjective"} [e2e-llm-inference-service] {"level":"info","ts":1785259945.6702995,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferenceobjective","controllerGroup":"llm-d.ai","controllerKind":"InferenceObjective","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1785259945.6702886,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"pod","controllerGroup":"","controllerKind":"Pod","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1785259945.7706294,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool"} [e2e-llm-inference-service] {"level":"info","ts":1785259945.7706745,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1785259945.77082,"caller":"controller/inferencepool_reconciler.go:46","msg":"Reconciling InferencePool","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"auth-enabled-test-inference-pool","namespace":"e2e-test-llm-auth-enabled-requires-token-d4ed2c08"},"namespace":"e2e-test-llm-auth-enabled-requires-token-d4ed2c08","name":"auth-enabled-test-inference-pool","reconcileID":"a08854d7-1cdd-4532-adcb-ce6dea0549ce"} [e2e-llm-inference-service] {"level":"info","ts":1785259957.9367824,"caller":"controller/inferencepool_reconciler.go:46","msg":"Reconciling InferencePool","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"auth-enabled-test-inference-pool","namespace":"e2e-test-llm-auth-enabled-requires-token-d4ed2c08"},"namespace":"e2e-test-llm-auth-enabled-requires-token-d4ed2c08","name":"auth-enabled-test-inference-pool","reconcileID":"7fafd5f7-275f-438b-a010-243d5ade5c1a"} [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: auth-enabled-test-epp-service [e2e-llm-inference-service] namespace: e2e-test-llm-auth-enabled-requires-token-d4ed2c08 [e2e-llm-inference-service] uid: 5ffd3f9b-3ab0-485d-b751-8eef9250ef1a [e2e-llm-inference-service] resourceVersion: '26133' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:32:25Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-28T17:32:57Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T17:32:57Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - ip: 10.132.0.37 [e2e-llm-inference-service] nodeName: ip-10-0-135-188.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-auth-enabled-requires-token-d4ed2c08 [e2e-llm-inference-service] name: auth-enabled-test-kserve-router-scheduler-5dbb5856d8-hgkl8 [e2e-llm-inference-service] uid: af04fa4a-00ab-4bd7-8cd9-61a7fc324cdc [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: auth-enabled-test-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-auth-enabled-requires-token-d4ed2c08 [e2e-llm-inference-service] uid: d5cf92af-3993-486c-a9b7-8484801abf05 [e2e-llm-inference-service] resourceVersion: '25815' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:32:24Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-28T17:32:24Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T17:32:29Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - notReadyAddresses: [e2e-llm-inference-service] - ip: 10.133.0.46 [e2e-llm-inference-service] nodeName: ip-10-0-140-52.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-auth-enabled-requires-token-d4ed2c08 [e2e-llm-inference-service] name: auth-enabled-test-kserve-766449dfcf-hn46v [e2e-llm-inference-service] uid: 0cb87bd2-bbb9-4adf-901e-ec29b1b15f19 [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: auth-enabled-test-kserve-766449dfcf-hn46v [e2e-llm-inference-service] generateName: auth-enabled-test-kserve-766449dfcf- [e2e-llm-inference-service] namespace: e2e-test-llm-auth-enabled-requires-token-d4ed2c08 [e2e-llm-inference-service] uid: 0cb87bd2-bbb9-4adf-901e-ec29b1b15f19 [e2e-llm-inference-service] resourceVersion: '25814' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:32:24Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: 766449dfcf [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.133.0.46/23"],"mac_address":"0a:58:0a:85:00:2e","gateway_ips":["10.133.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.133.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.133.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.133.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.133.0.1"}],"ip_address":"10.133.0.46/23","gateway_ip":"10.133.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.133.0.46\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:85:00:2e\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: auth-enabled-test-kserve-766449dfcf [e2e-llm-inference-service] uid: 3a78c1fd-42d9-40a4-998d-92b1ea1a6970 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-140-52 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T17:32:24Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T17:32:24Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"3a78c1fd-42d9-40a4-998d-92b1ea1a6970"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_HIGH_PERFORMANCE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TOKIO_WORKER_THREADS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T17:32:25Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T17:32:29Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:initContainerStatuses: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.133.0.46"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: auth-enabled-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kube-api-access-8xkhq [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da [e2e-llm-inference-service] args: [e2e-llm-inference-service] - hf://facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: TOKIO_WORKER_THREADS [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: HF_HUB_ENABLE_HF_TRANSFER [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_HIGH_PERFORMANCE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_NUM_CONCURRENT_RANGE_GETS [e2e-llm-inference-service] value: '8' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-8xkhq [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000760000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to infer\ [e2e-llm-inference-service] \ RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null\n\ [e2e-llm-inference-service] \ grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/* 2>/dev/null\n\ [e2e-llm-inference-service] \n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"$hca_dir\"\ [e2e-llm-inference-service] \ ]; then\n hca_name=$(basename \"$hca_dir\")\n port_state_file=\"\ [e2e-llm-inference-service] $hca_dir/ports/1/state\" # Assume port 1\n type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\ [e2e-llm-inference-service] \n\n echo \"[Infer RoCE] Check if the port state file ${port_state_file}\ [e2e-llm-inference-service] \ exists and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] &&\ [e2e-llm-inference-service] \ grep -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found active\ [e2e-llm-inference-service] \ HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n else\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Skipping inactive or down HCA: $hca_name\"\ [e2e-llm-inference-service] \n fi\n fi\n done\n\n # Check if we found any active HCAs\n\ [e2e-llm-inference-service] \ if [ ${#active_hcas[@]} -gt 0 ]; then\n # Join the array elements with\ [e2e-llm-inference-service] \ a comma\n hca_port_pairs=()\n for hca in \"${active_hcas[@]}\";\ [e2e-llm-inference-service] \ do\n hca_port_pairs+=(\"${hca}:1\")\n done\n\n active_hca_list=$(IFS=,;\ [e2e-llm-inference-service] \ echo \"${active_hcas[*]}\")\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\"\ [e2e-llm-inference-service] )\n echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n\ [e2e-llm-inference-service] \ export UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found. NCCL_IB_HCA\ [e2e-llm-inference-service] \ will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Finding GID_INDEX for each active HCA (SR-IOV compatible)...\"\ [e2e-llm-inference-service] \n\n # For SR-IOV environments, find the most common IPv4 RoCE v2 GID index\ [e2e-llm-inference-service] \ across all HCAs\n declare -A gid_index_count\n declare -A hca_gid_index\n\ [e2e-llm-inference-service] \n for hca_name in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Processing HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for\ [e2e-llm-inference-service] \ this HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"$tpath\"\ [e2e-llm-inference-service] \ 2>/dev/null; then\n idx=$(basename \"$tpath\")\n \ [e2e-llm-inference-service] \ gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n \ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo \"\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Found IPv4 RoCE v2 GID for ${hca_name}:\ [e2e-llm-inference-service] \ index=${idx}, gid=${gid_value}\"\n hca_gid_index[\"${hca_name}\"\ [e2e-llm-inference-service] ]=\"${idx}\"\n gid_index_count[\"${idx}\"]=$((${gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]} + 1))\n break # Use first found IPv4 GID per\ [e2e-llm-inference-service] \ HCA\n fi\n fi\n done\n done\n\n\ [e2e-llm-inference-service] \ # Find the most common GID index (most likely to be consistent across\ [e2e-llm-inference-service] \ nodes)\n best_gid_index=\"\"\n max_count=0\n for idx in \"\ [e2e-llm-inference-service] ${!gid_index_count[@]}\"; do\n count=${gid_index_count[\"${idx}\"]}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n \ [e2e-llm-inference-service] \ if [ $count -gt $max_count ]; then\n max_count=$count\n\ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n #\ [e2e-llm-inference-service] \ Use deterministic fallback if tied - prefer index 3 (SR-IOV standard)\n \ [e2e-llm-inference-service] \ if [ ${#gid_index_count[@]} -gt 1 ]; then\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Multiple GID indices found, selecting most common: ${best_gid_index}\"\n \ [e2e-llm-inference-service] \ # If there's a tie, prefer index 3 as it's most common in SR-IOV setups\n\ [e2e-llm-inference-service] \ if [ -n \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\"\ [e2e-llm-inference-service] \ -eq \"$max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for NCCL,\ [e2e-llm-inference-service] \ NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR: No valid\ [e2e-llm-inference-service] \ IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any HCA.\"\n \ [e2e-llm-inference-service] \ fi\n else\n echo \"[Infer RoCE] No active HCAs found, skipping GID_INDEX\ [e2e-llm-inference-service] \ inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints landed in vLLM\ [e2e-llm-inference-service] \ 0.16.0 (vllm-project/vllm#30011).\n# Older versions still need the blanket\ [e2e-llm-inference-service] \ --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+ ]] &&\ [e2e-llm-inference-service] \ [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort -V | head\ [e2e-llm-inference-service] \ -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout 40\"\ [e2e-llm-inference-service] \nfi\n\n# --kv-transfer-config with OffloadingConnector requires vLLM 0.22.0+\ [e2e-llm-inference-service] \ (vllm-project/vllm#40020).\nKV_TRANSFER_ARGS=\"\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.22.0\" ]; then\n if [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\" != *\"\ [e2e-llm-inference-service] --kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"* ]] &&\ [e2e-llm-inference-service] \ [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n KV_TRANSFER_ARGS=\"\"\ [e2e-llm-inference-service] \n fi\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name \"\ [e2e-llm-inference-service] facebook/opt-125m\" \"publishers/e2e-test-llm-auth-enabled-requires-token-d4ed2c08/models/facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8000 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n ${KV_TRANSFER_ARGS} \\\n \\\n --enable-ssl-refresh \\\n --ssl-certfile\ [e2e-llm-inference-service] \ /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-8xkhq [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000760000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: default [e2e-llm-inference-service] serviceAccount: default [e2e-llm-inference-service] nodeName: ip-10-0-140-52.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c28,c2 [e2e-llm-inference-service] fsGroup: 1000760000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-xrt2x [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Pending [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T17:32:29Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T17:32:24Z' [e2e-llm-inference-service] reason: ContainersNotInitialized [e2e-llm-inference-service] message: 'containers with incomplete status: [storage-initializer]' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T17:32:24Z' [e2e-llm-inference-service] reason: ContainersNotReady [e2e-llm-inference-service] message: 'containers with unready status: [main]' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T17:32:24Z' [e2e-llm-inference-service] reason: ContainersNotReady [e2e-llm-inference-service] message: 'containers with unready status: [main]' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T17:32:24Z' [e2e-llm-inference-service] hostIP: 10.0.140.52 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.140.52 [e2e-llm-inference-service] podIP: 10.133.0.46 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.133.0.46 [e2e-llm-inference-service] startTime: '2026-07-28T17:32:24Z' [e2e-llm-inference-service] initContainerStatuses: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-07-28T17:32:28Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: false [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da [e2e-llm-inference-service] imageID: quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da [e2e-llm-inference-service] containerID: cri-o://1b96e55b3acb3fcab66126d92e868cc484e4db0136a0a471236fa1d5f236dc4b [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-8xkhq [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000760000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000760000 [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] waiting: [e2e-llm-inference-service] reason: PodInitializing [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: false [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] imageID: '' [e2e-llm-inference-service] started: false [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-8xkhq [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: auth-enabled-test-kserve-router-scheduler-5dbb5856d8-hgkl8 [e2e-llm-inference-service] generateName: auth-enabled-test-kserve-router-scheduler-5dbb5856d8- [e2e-llm-inference-service] namespace: e2e-test-llm-auth-enabled-requires-token-d4ed2c08 [e2e-llm-inference-service] uid: af04fa4a-00ab-4bd7-8cd9-61a7fc324cdc [e2e-llm-inference-service] resourceVersion: '26131' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:32:24Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 5dbb5856d8 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.10.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.132.0.37/23"],"mac_address":"0a:58:0a:84:00:25","gateway_ips":["10.132.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.132.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.132.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.132.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.132.0.1"}],"ip_address":"10.132.0.37/23","gateway_ip":"10.132.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.132.0.37\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:84:00:25\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: auth-enabled-test-kserve-router-scheduler-5dbb5856d8 [e2e-llm-inference-service] uid: ccffd31a-744e-442d-b6ed-9b58e383bbfb [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-135-188 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T17:32:24Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T17:32:24Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"ccffd31a-744e-442d-b6ed-9b58e383bbfb"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T17:32:25Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T17:32:57Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.132.0.37"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: auth-enabled-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kube-api-access-7k7np [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - auth-enabled-test-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - e2e-test-llm-auth-enabled-requires-token-d4ed2c08 [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: llm-d.ai/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n- type:\ [e2e-llm-inference-service] \ single-profile-handler\n- type: queue-scorer\n- type: kv-cache-utilization-scorer\n\ [e2e-llm-inference-service] - type: prefix-cache-scorer\n- type: no-hit-lru-scorer\n- type: max-score-picker\n\ [e2e-llm-inference-service] - parameters:\n scheme: https\n type: metrics-data-source\nschedulingProfiles:\n\ [e2e-llm-inference-service] - name: default\n plugins:\n - pluginRef: queue-scorer\n weight: 2\n -\ [e2e-llm-inference-service] \ pluginRef: kv-cache-utilization-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n\ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: no-hit-lru-scorer\n weight: 2\n - pluginRef:\ [e2e-llm-inference-service] \ max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kube-api-access-7k7np [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000760000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: auth-enabled-test-epp-sa [e2e-llm-inference-service] serviceAccount: auth-enabled-test-epp-sa [e2e-llm-inference-service] nodeName: ip-10-0-135-188.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c28,c2 [e2e-llm-inference-service] fsGroup: 1000760000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-xrt2x [e2e-llm-inference-service] - name: auth-enabled-test-epp-sa-dockercfg-jkgsn [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T17:32:26Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T17:32:24Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T17:32:57Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T17:32:57Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T17:32:24Z' [e2e-llm-inference-service] hostIP: 10.0.135.188 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.135.188 [e2e-llm-inference-service] podIP: 10.132.0.37 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.132.0.37 [e2e-llm-inference-service] startTime: '2026-07-28T17:32:24Z' [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-07-28T17:32:25Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-llm-d-router-endpoint-picker@sha256:7fd1d61cb5505e026eebab0b9653758779c3dee241e37bea92a0bab961a48c50 [e2e-llm-inference-service] containerID: cri-o://2e452ac0ebc40805ca1f6f293ec77e10bb8da89ea22f04a4bf42bbc6bb6036bf [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-7k7np [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000760000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000760000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: auth-enabled-test-epp-sa [e2e-llm-inference-service] namespace: e2e-test-llm-auth-enabled-requires-token-d4ed2c08 [e2e-llm-inference-service] uid: d5808660-d6fb-44ef-9880-f3f8e9ab3076 [e2e-llm-inference-service] resourceVersion: '25592' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:32:24Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] openshift.io/internal-registry-pull-secret-ref: auth-enabled-test-epp-sa-dockercfg-jkgsn [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: auth-enabled-test [e2e-llm-inference-service] uid: 3f4fbd10-5888-4617-9e6f-f440212e4c59 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: openshift.io/image-registry-pull-secrets_service-account-controller [e2e-llm-inference-service] operation: Apply [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T17:32:24Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:imagePullSecrets: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:openshift.io/internal-registry-pull-secret-ref: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] k:{"name":"auth-enabled-test-epp-sa-dockercfg-jkgsn"}: {} [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T17:32:24Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"3f4fbd10-5888-4617-9e6f-f440212e4c59"}: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"default-dockercfg-xrt2x"}: {} [e2e-llm-inference-service] k:{"name":"seaweedfs-s3-creds"}: {} [e2e-llm-inference-service] secrets: [e2e-llm-inference-service] - name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: default-dockercfg-xrt2x [e2e-llm-inference-service] - name: auth-enabled-test-epp-sa-dockercfg-jkgsn [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-xrt2x [e2e-llm-inference-service] - name: auth-enabled-test-epp-sa-dockercfg-jkgsn [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: ServiceAccount [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: auth-enabled-test-epp-service [e2e-llm-inference-service] namespace: e2e-test-llm-auth-enabled-requires-token-d4ed2c08 [e2e-llm-inference-service] uid: 1a6750ad-5c1b-45b0-a365-c90f0aea6313 [e2e-llm-inference-service] resourceVersion: '25622' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:32:25Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: auth-enabled-test [e2e-llm-inference-service] uid: 3f4fbd10-5888-4617-9e6f-f440212e4c59 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T17:32:25Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"3f4fbd10-5888-4617-9e6f-f440212e4c59"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] targetPort: grpc [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] targetPort: grpc-health [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] targetPort: metrics [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] targetPort: zmq [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] clusterIP: 172.31.116.188 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.116.188 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: auth-enabled-test-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-auth-enabled-requires-token-d4ed2c08 [e2e-llm-inference-service] uid: d7d3ddde-37fe-487a-b148-3b13279c1136 [e2e-llm-inference-service] resourceVersion: '25585' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:32:24Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: auth-enabled-test [e2e-llm-inference-service] uid: 3f4fbd10-5888-4617-9e6f-f440212e4c59 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T17:32:24Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"3f4fbd10-5888-4617-9e6f-f440212e4c59"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] targetPort: 8000 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] clusterIP: 172.31.172.168 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.172.168 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: auth-enabled-test-kserve [e2e-llm-inference-service] namespace: e2e-test-llm-auth-enabled-requires-token-d4ed2c08 [e2e-llm-inference-service] uid: 7d4a2853-4b66-442e-afaa-46117177eb96 [e2e-llm-inference-service] resourceVersion: '31809' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:32:24Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: auth-enabled-test [e2e-llm-inference-service] uid: 3f4fbd10-5888-4617-9e6f-f440212e4c59 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T17:32:24Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"3f4fbd10-5888-4617-9e6f-f440212e4c59"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:rollingUpdate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:maxSurge: {} [e2e-llm-inference-service] f:maxUnavailable: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_HIGH_PERFORMANCE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TOKIO_WORKER_THREADS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T17:42:25Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:unavailableReplicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: auth-enabled-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da [e2e-llm-inference-service] args: [e2e-llm-inference-service] - hf://facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: TOKIO_WORKER_THREADS [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: HF_HUB_ENABLE_HF_TRANSFER [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_HIGH_PERFORMANCE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_NUM_CONCURRENT_RANGE_GETS [e2e-llm-inference-service] value: '8' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n # Check if\ [e2e-llm-inference-service] \ we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ # Join the array elements with a comma\n hca_port_pairs=()\n \ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n hca_port_pairs+=(\"\ [e2e-llm-inference-service] ${hca}:1\")\n done\n\n active_hca_list=$(IFS=,; echo \"${active_hcas[*]}\"\ [e2e-llm-inference-service] )\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\n# --kv-transfer-config with OffloadingConnector requires vLLM\ [e2e-llm-inference-service] \ 0.22.0+ (vllm-project/vllm#40020).\nKV_TRANSFER_ARGS=\"\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.22.0\" ]; then\n if [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\" !=\ [e2e-llm-inference-service] \ *\"--kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"\ [e2e-llm-inference-service] * ]] && [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n KV_TRANSFER_ARGS=\"\ [e2e-llm-inference-service] \"\n fi\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \"publishers/e2e-test-llm-auth-enabled-requires-token-d4ed2c08/models/facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8000 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n ${KV_TRANSFER_ARGS} \\\n \\\n --enable-ssl-refresh \\\n --ssl-certfile\ [e2e-llm-inference-service] \ /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: RollingUpdate [e2e-llm-inference-service] rollingUpdate: [e2e-llm-inference-service] maxUnavailable: 25% [e2e-llm-inference-service] maxSurge: 25% [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] unavailableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastUpdateTime: '2026-07-28T17:32:24Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T17:32:24Z' [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastUpdateTime: '2026-07-28T17:42:25Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T17:42:25Z' [e2e-llm-inference-service] reason: ProgressDeadlineExceeded [e2e-llm-inference-service] message: ReplicaSet "auth-enabled-test-kserve-766449dfcf" has timed out progressing. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: auth-enabled-test-kserve-router-scheduler [e2e-llm-inference-service] namespace: e2e-test-llm-auth-enabled-requires-token-d4ed2c08 [e2e-llm-inference-service] uid: 43fcf506-6961-4201-aac8-07c1cf28eedf [e2e-llm-inference-service] resourceVersion: '26135' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:32:24Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: auth-enabled-test [e2e-llm-inference-service] uid: 3f4fbd10-5888-4617-9e6f-f440212e4c59 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T17:32:24Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"3f4fbd10-5888-4617-9e6f-f440212e4c59"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T17:32:57Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.10.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: auth-enabled-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - auth-enabled-test-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - e2e-test-llm-auth-enabled-requires-token-d4ed2c08 [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: llm-d.ai/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n-\ [e2e-llm-inference-service] \ type: single-profile-handler\n- type: queue-scorer\n- type: kv-cache-utilization-scorer\n\ [e2e-llm-inference-service] - type: prefix-cache-scorer\n- type: no-hit-lru-scorer\n- type: max-score-picker\n\ [e2e-llm-inference-service] - parameters:\n scheme: https\n type: metrics-data-source\nschedulingProfiles:\n\ [e2e-llm-inference-service] - name: default\n plugins:\n - pluginRef: queue-scorer\n weight: 2\n\ [e2e-llm-inference-service] \ - pluginRef: kv-cache-utilization-scorer\n weight: 2\n - pluginRef:\ [e2e-llm-inference-service] \ prefix-cache-scorer\n weight: 3\n - pluginRef: no-hit-lru-scorer\n\ [e2e-llm-inference-service] \ weight: 2\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: auth-enabled-test-epp-sa [e2e-llm-inference-service] serviceAccount: auth-enabled-test-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: Recreate [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-28T17:32:57Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T17:32:57Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-28T17:32:57Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T17:32:24Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "auth-enabled-test-kserve-router-scheduler-5dbb5856d8" has [e2e-llm-inference-service] successfully progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: auth-enabled-test-kserve-766449dfcf [e2e-llm-inference-service] namespace: e2e-test-llm-auth-enabled-requires-token-d4ed2c08 [e2e-llm-inference-service] uid: 3a78c1fd-42d9-40a4-998d-92b1ea1a6970 [e2e-llm-inference-service] resourceVersion: '25582' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:32:24Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: 766449dfcf [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '2' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: auth-enabled-test-kserve [e2e-llm-inference-service] uid: 7d4a2853-4b66-442e-afaa-46117177eb96 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T17:32:24Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7d4a2853-4b66-442e-afaa-46117177eb96"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_HIGH_PERFORMANCE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TOKIO_WORKER_THREADS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T17:32:24Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: 766449dfcf [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: 766449dfcf [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: auth-enabled-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da [e2e-llm-inference-service] args: [e2e-llm-inference-service] - hf://facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: TOKIO_WORKER_THREADS [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: HF_HUB_ENABLE_HF_TRANSFER [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_HIGH_PERFORMANCE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_NUM_CONCURRENT_RANGE_GETS [e2e-llm-inference-service] value: '8' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n # Check if\ [e2e-llm-inference-service] \ we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ # Join the array elements with a comma\n hca_port_pairs=()\n \ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n hca_port_pairs+=(\"\ [e2e-llm-inference-service] ${hca}:1\")\n done\n\n active_hca_list=$(IFS=,; echo \"${active_hcas[*]}\"\ [e2e-llm-inference-service] )\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\n# --kv-transfer-config with OffloadingConnector requires vLLM\ [e2e-llm-inference-service] \ 0.22.0+ (vllm-project/vllm#40020).\nKV_TRANSFER_ARGS=\"\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.22.0\" ]; then\n if [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\" !=\ [e2e-llm-inference-service] \ *\"--kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"\ [e2e-llm-inference-service] * ]] && [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n KV_TRANSFER_ARGS=\"\ [e2e-llm-inference-service] \"\n fi\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \"publishers/e2e-test-llm-auth-enabled-requires-token-d4ed2c08/models/facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8000 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n ${KV_TRANSFER_ARGS} \\\n \\\n --enable-ssl-refresh \\\n --ssl-certfile\ [e2e-llm-inference-service] \ /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: auth-enabled-test-kserve-router-scheduler-5dbb5856d8 [e2e-llm-inference-service] namespace: e2e-test-llm-auth-enabled-requires-token-d4ed2c08 [e2e-llm-inference-service] uid: ccffd31a-744e-442d-b6ed-9b58e383bbfb [e2e-llm-inference-service] resourceVersion: '26134' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:32:24Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 5dbb5856d8 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: auth-enabled-test-kserve-router-scheduler [e2e-llm-inference-service] uid: 43fcf506-6961-4201-aac8-07c1cf28eedf [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T17:32:24Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"43fcf506-6961-4201-aac8-07c1cf28eedf"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T17:32:57Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 5dbb5856d8 [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 5dbb5856d8 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.10.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: auth-enabled-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - auth-enabled-test-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - e2e-test-llm-auth-enabled-requires-token-d4ed2c08 [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: llm-d.ai/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n-\ [e2e-llm-inference-service] \ type: single-profile-handler\n- type: queue-scorer\n- type: kv-cache-utilization-scorer\n\ [e2e-llm-inference-service] - type: prefix-cache-scorer\n- type: no-hit-lru-scorer\n- type: max-score-picker\n\ [e2e-llm-inference-service] - parameters:\n scheme: https\n type: metrics-data-source\nschedulingProfiles:\n\ [e2e-llm-inference-service] - name: default\n plugins:\n - pluginRef: queue-scorer\n weight: 2\n\ [e2e-llm-inference-service] \ - pluginRef: kv-cache-utilization-scorer\n weight: 2\n - pluginRef:\ [e2e-llm-inference-service] \ prefix-cache-scorer\n weight: 3\n - pluginRef: no-hit-lru-scorer\n\ [e2e-llm-inference-service] \ weight: 2\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: auth-enabled-test-epp-sa [e2e-llm-inference-service] serviceAccount: auth-enabled-test-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: auth-enabled-test-epp-rb [e2e-llm-inference-service] namespace: e2e-test-llm-auth-enabled-requires-token-d4ed2c08 [e2e-llm-inference-service] uid: c801fd61-79f2-4ca2-88ba-2237b6d0d7c4 [e2e-llm-inference-service] resourceVersion: '25603' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:32:24Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: auth-enabled-test [e2e-llm-inference-service] uid: 3f4fbd10-5888-4617-9e6f-f440212e4c59 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T17:32:24Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"3f4fbd10-5888-4617-9e6f-f440212e4c59"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: auth-enabled-test-epp-sa [e2e-llm-inference-service] namespace: e2e-test-llm-auth-enabled-requires-token-d4ed2c08 [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] apiGroup: rbac.authorization.k8s.io [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] name: auth-enabled-test-epp-role [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: auth-enabled-test-epp-role [e2e-llm-inference-service] namespace: e2e-test-llm-auth-enabled-requires-token-d4ed2c08 [e2e-llm-inference-service] uid: 91c3c810-de48-4b8c-9e37-74720aa29d2e [e2e-llm-inference-service] resourceVersion: '25599' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:32:24Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: auth-enabled-test [e2e-llm-inference-service] uid: 3f4fbd10-5888-4617-9e6f-f440212e4c59 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T17:32:24Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"3f4fbd10-5888-4617-9e6f-f440212e4c59"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - llm-d.ai [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] - create [e2e-llm-inference-service] - update [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - delete [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: auth-enabled-test-epp-service-c54jv [e2e-llm-inference-service] generateName: auth-enabled-test-epp-service- [e2e-llm-inference-service] namespace: e2e-test-llm-auth-enabled-requires-token-d4ed2c08 [e2e-llm-inference-service] uid: c3314cda-0b68-424d-b0be-852de4f28d10 [e2e-llm-inference-service] resourceVersion: '26132' [e2e-llm-inference-service] generation: 3 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:32:25Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: auth-enabled-test-epp-service [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-28T17:32:57Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: auth-enabled-test-epp-service [e2e-llm-inference-service] uid: 1a6750ad-5c1b-45b0-a365-c90f0aea6313 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T17:32:57Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"1a6750ad-5c1b-45b0-a365-c90f0aea6313"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.132.0.37 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-auth-enabled-requires-token-d4ed2c08 [e2e-llm-inference-service] name: auth-enabled-test-kserve-router-scheduler-5dbb5856d8-hgkl8 [e2e-llm-inference-service] uid: af04fa4a-00ab-4bd7-8cd9-61a7fc324cdc [e2e-llm-inference-service] nodeName: ip-10-0-135-188.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: auth-enabled-test-kserve-workload-svc-mt8wf [e2e-llm-inference-service] generateName: auth-enabled-test-kserve-workload-svc- [e2e-llm-inference-service] namespace: e2e-test-llm-auth-enabled-requires-token-d4ed2c08 [e2e-llm-inference-service] uid: 478655c8-e597-4bd1-9a33-7e7eab409902 [e2e-llm-inference-service] resourceVersion: '25816' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:32:24Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: auth-enabled-test-kserve-workload-svc [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: auth-enabled-test-kserve-workload-svc [e2e-llm-inference-service] uid: d7d3ddde-37fe-487a-b148-3b13279c1136 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T17:32:29Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d7d3ddde-37fe-487a-b148-3b13279c1136"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.133.0.46 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: false [e2e-llm-inference-service] serving: false [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-auth-enabled-requires-token-d4ed2c08 [e2e-llm-inference-service] name: auth-enabled-test-kserve-766449dfcf-hn46v [e2e-llm-inference-service] uid: 0cb87bd2-bbb9-4adf-901e-ec29b1b15f19 [e2e-llm-inference-service] nodeName: ip-10-0-140-52.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: auth-enabled-test-epp-rb [e2e-llm-inference-service] namespace: e2e-test-llm-auth-enabled-requires-token-d4ed2c08 [e2e-llm-inference-service] uid: c801fd61-79f2-4ca2-88ba-2237b6d0d7c4 [e2e-llm-inference-service] resourceVersion: '25603' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:32:24Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: auth-enabled-test [e2e-llm-inference-service] uid: 3f4fbd10-5888-4617-9e6f-f440212e4c59 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T17:32:24Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"3f4fbd10-5888-4617-9e6f-f440212e4c59"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] userNames: [e2e-llm-inference-service] - system:serviceaccount:e2e-test-llm-auth-enabled-requires-token-d4ed2c08:auth-enabled-test-epp-sa [e2e-llm-inference-service] groupNames: null [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] namespace: e2e-test-llm-auth-enabled-requires-token-d4ed2c08 [e2e-llm-inference-service] name: auth-enabled-test-epp-sa [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] namespace: e2e-test-llm-auth-enabled-requires-token-d4ed2c08 [e2e-llm-inference-service] name: auth-enabled-test-epp-role [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: auth-enabled-test-epp-role [e2e-llm-inference-service] namespace: e2e-test-llm-auth-enabled-requires-token-d4ed2c08 [e2e-llm-inference-service] uid: 91c3c810-de48-4b8c-9e37-74720aa29d2e [e2e-llm-inference-service] resourceVersion: '25599' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:32:24Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: auth-enabled-test [e2e-llm-inference-service] uid: 3f4fbd10-5888-4617-9e6f-f440212e4c59 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T17:32:24Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"3f4fbd10-5888-4617-9e6f-f440212e4c59"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - llm-d.ai [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - create [e2e-llm-inference-service] - delete [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - update [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:32:25Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"3f4fbd10-5888-4617-9e6f-f440212e4c59"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T17:32:37Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-28T17:32:38Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-28T17:32:38Z' [e2e-llm-inference-service] name: auth-enabled-test-kserve-route [e2e-llm-inference-service] namespace: e2e-test-llm-auth-enabled-requires-token-d4ed2c08 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: auth-enabled-test [e2e-llm-inference-service] uid: 3f4fbd10-5888-4617-9e6f-f440212e4c59 [e2e-llm-inference-service] resourceVersion: '25934' [e2e-llm-inference-service] uid: e05b327b-453a-4f7f-b321-800a53307e3a [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: auth-enabled-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: auth-enabled-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: auth-enabled-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: auth-enabled-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: auth-enabled-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-auth-enabled-requires-token-d4ed2c08/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-auth-enabled-requires-token-d4ed2c08/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-auth-enabled-requires-token-d4ed2c08/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-auth-enabled-requires-token-d4ed2c08/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-auth-enabled-requires-token-d4ed2c08/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-auth-enabled-requires-token-d4ed2c08/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-auth-enabled-requires-token-d4ed2c08/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-auth-enabled-requires-token-d4ed2c08/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: auth-enabled-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-auth-enabled-requires-token-d4ed2c08/models/facebook/opt-125m/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: auth-enabled-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-auth-enabled-requires-token-d4ed2c08/models/facebook/opt-125m/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: auth-enabled-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-auth-enabled-requires-token-d4ed2c08/models/facebook/opt-125m/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: auth-enabled-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-auth-enabled-requires-token-d4ed2c08/models/facebook/opt-125m/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: auth-enabled-test-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-auth-enabled-requires-token-d4ed2c08/models/facebook/opt-125m [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: auth-enabled-test-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: auth-enabled-test-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-auth-enabled-requires-token-d4ed2c08/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T17:32:25Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T17:32:38Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T17:32:26Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:32:25Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"3f4fbd10-5888-4617-9e6f-f440212e4c59"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T17:32:37Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-28T17:32:38Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-28T17:32:38Z' [e2e-llm-inference-service] name: auth-enabled-test-kserve-route [e2e-llm-inference-service] namespace: e2e-test-llm-auth-enabled-requires-token-d4ed2c08 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: auth-enabled-test [e2e-llm-inference-service] uid: 3f4fbd10-5888-4617-9e6f-f440212e4c59 [e2e-llm-inference-service] resourceVersion: '25934' [e2e-llm-inference-service] uid: e05b327b-453a-4f7f-b321-800a53307e3a [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: auth-enabled-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: auth-enabled-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: auth-enabled-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: auth-enabled-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: auth-enabled-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-auth-enabled-requires-token-d4ed2c08/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-auth-enabled-requires-token-d4ed2c08/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-auth-enabled-requires-token-d4ed2c08/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-auth-enabled-requires-token-d4ed2c08/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-auth-enabled-requires-token-d4ed2c08/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-auth-enabled-requires-token-d4ed2c08/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-auth-enabled-requires-token-d4ed2c08/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-auth-enabled-requires-token-d4ed2c08/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: auth-enabled-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-auth-enabled-requires-token-d4ed2c08/models/facebook/opt-125m/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: auth-enabled-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-auth-enabled-requires-token-d4ed2c08/models/facebook/opt-125m/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: auth-enabled-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-auth-enabled-requires-token-d4ed2c08/models/facebook/opt-125m/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: auth-enabled-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-auth-enabled-requires-token-d4ed2c08/models/facebook/opt-125m/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: auth-enabled-test-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-auth-enabled-requires-token-d4ed2c08/models/facebook/opt-125m [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: auth-enabled-test-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: auth-enabled-test-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-auth-enabled-requires-token-d4ed2c08/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T17:32:25Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T17:32:38Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T17:32:26Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:32:25Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"3f4fbd10-5888-4617-9e6f-f440212e4c59"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:endpointPickerRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:number: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:matchLabels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPorts: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T17:32:25Z' [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-28T17:32:37Z' [e2e-llm-inference-service] name: auth-enabled-test-inference-pool [e2e-llm-inference-service] namespace: e2e-test-llm-auth-enabled-requires-token-d4ed2c08 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: auth-enabled-test [e2e-llm-inference-service] uid: 3f4fbd10-5888-4617-9e6f-f440212e4c59 [e2e-llm-inference-service] resourceVersion: '25916' [e2e-llm-inference-service] uid: bebcfa19-7005-4550-86a2-7502598e743f [e2e-llm-inference-service] spec: [e2e-llm-inference-service] appProtocol: http [e2e-llm-inference-service] endpointPickerRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: auth-enabled-test-epp-service [e2e-llm-inference-service] port: [e2e-llm-inference-service] number: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPorts: [e2e-llm-inference-service] - number: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T17:32:37Z' [e2e-llm-inference-service] message: Referenced by an HTTPRoute accepted by the parentRef Gateway [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T17:32:37Z' [e2e-llm-inference-service] message: Referenced ExtensionRef resolved successfully [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: networking.istio.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:32:25Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"3f4fbd10-5888-4617-9e6f-f440212e4c59"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T17:32:25Z' [e2e-llm-inference-service] name: auth-enabled-test-kserve-scheduler [e2e-llm-inference-service] namespace: e2e-test-llm-auth-enabled-requires-token-d4ed2c08 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: auth-enabled-test [e2e-llm-inference-service] uid: 3f4fbd10-5888-4617-9e6f-f440212e4c59 [e2e-llm-inference-service] resourceVersion: '25669' [e2e-llm-inference-service] uid: ef78f51a-5a8b-4c7c-bf99-b80bfddcd2d0 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: auth-enabled-test-epp-service.e2e-test-llm-auth-enabled-requires-token-d4ed2c08.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: auth-enabled-test-epp-service.e2e-test-llm-auth-enabled-requires-token-d4ed2c08.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:32:38Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"3f4fbd10-5888-4617-9e6f-f440212e4c59"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T17:32:38Z' [e2e-llm-inference-service] name: auth-enabled-test-kserve-shadow-svc [e2e-llm-inference-service] namespace: e2e-test-llm-auth-enabled-requires-token-d4ed2c08 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: auth-enabled-test [e2e-llm-inference-service] uid: 3f4fbd10-5888-4617-9e6f-f440212e4c59 [e2e-llm-inference-service] resourceVersion: '25926' [e2e-llm-inference-service] uid: 47dcf1d8-563a-435e-87ae-27c3dc86164a [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: auth-enabled-test-inference-pool-ip-0e6361b2.e2e-test-llm-auth-enabled-requires-token-d4ed2c08.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: auth-enabled-test-kserve-workload-svc.e2e-test-llm-auth-enabled-requires-token-d4ed2c08.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:32:25Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"3f4fbd10-5888-4617-9e6f-f440212e4c59"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T17:32:25Z' [e2e-llm-inference-service] name: auth-enabled-test-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-auth-enabled-requires-token-d4ed2c08 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: auth-enabled-test [e2e-llm-inference-service] uid: 3f4fbd10-5888-4617-9e6f-f440212e4c59 [e2e-llm-inference-service] resourceVersion: '25686' [e2e-llm-inference-service] uid: bc1093f1-073e-47ea-ad96-f958fe7aabae [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: auth-enabled-test-kserve-workload-svc.e2e-test-llm-auth-enabled-requires-token-d4ed2c08.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: auth-enabled-test-kserve-workload-svc.e2e-test-llm-auth-enabled-requires-token-d4ed2c08.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:32:25Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"3f4fbd10-5888-4617-9e6f-f440212e4c59"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T17:32:25Z' [e2e-llm-inference-service] name: auth-enabled-test-kserve-scheduler [e2e-llm-inference-service] namespace: e2e-test-llm-auth-enabled-requires-token-d4ed2c08 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: auth-enabled-test [e2e-llm-inference-service] uid: 3f4fbd10-5888-4617-9e6f-f440212e4c59 [e2e-llm-inference-service] resourceVersion: '25669' [e2e-llm-inference-service] uid: ef78f51a-5a8b-4c7c-bf99-b80bfddcd2d0 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: auth-enabled-test-epp-service.e2e-test-llm-auth-enabled-requires-token-d4ed2c08.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: auth-enabled-test-epp-service.e2e-test-llm-auth-enabled-requires-token-d4ed2c08.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:32:38Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"3f4fbd10-5888-4617-9e6f-f440212e4c59"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T17:32:38Z' [e2e-llm-inference-service] name: auth-enabled-test-kserve-shadow-svc [e2e-llm-inference-service] namespace: e2e-test-llm-auth-enabled-requires-token-d4ed2c08 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: auth-enabled-test [e2e-llm-inference-service] uid: 3f4fbd10-5888-4617-9e6f-f440212e4c59 [e2e-llm-inference-service] resourceVersion: '25926' [e2e-llm-inference-service] uid: 47dcf1d8-563a-435e-87ae-27c3dc86164a [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: auth-enabled-test-inference-pool-ip-0e6361b2.e2e-test-llm-auth-enabled-requires-token-d4ed2c08.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: auth-enabled-test-kserve-workload-svc.e2e-test-llm-auth-enabled-requires-token-d4ed2c08.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:32:25Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"3f4fbd10-5888-4617-9e6f-f440212e4c59"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T17:32:25Z' [e2e-llm-inference-service] name: auth-enabled-test-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-auth-enabled-requires-token-d4ed2c08 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: auth-enabled-test [e2e-llm-inference-service] uid: 3f4fbd10-5888-4617-9e6f-f440212e4c59 [e2e-llm-inference-service] resourceVersion: '25686' [e2e-llm-inference-service] uid: bc1093f1-073e-47ea-ad96-f958fe7aabae [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: auth-enabled-test-kserve-workload-svc.e2e-test-llm-auth-enabled-requires-token-d4ed2c08.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: auth-enabled-test-kserve-workload-svc.e2e-test-llm-auth-enabled-requires-token-d4ed2c08.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:32:25Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"3f4fbd10-5888-4617-9e6f-f440212e4c59"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T17:32:25Z' [e2e-llm-inference-service] name: auth-enabled-test-kserve-scheduler [e2e-llm-inference-service] namespace: e2e-test-llm-auth-enabled-requires-token-d4ed2c08 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: auth-enabled-test [e2e-llm-inference-service] uid: 3f4fbd10-5888-4617-9e6f-f440212e4c59 [e2e-llm-inference-service] resourceVersion: '25669' [e2e-llm-inference-service] uid: ef78f51a-5a8b-4c7c-bf99-b80bfddcd2d0 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: auth-enabled-test-epp-service.e2e-test-llm-auth-enabled-requires-token-d4ed2c08.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: auth-enabled-test-epp-service.e2e-test-llm-auth-enabled-requires-token-d4ed2c08.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:32:38Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"3f4fbd10-5888-4617-9e6f-f440212e4c59"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T17:32:38Z' [e2e-llm-inference-service] name: auth-enabled-test-kserve-shadow-svc [e2e-llm-inference-service] namespace: e2e-test-llm-auth-enabled-requires-token-d4ed2c08 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: auth-enabled-test [e2e-llm-inference-service] uid: 3f4fbd10-5888-4617-9e6f-f440212e4c59 [e2e-llm-inference-service] resourceVersion: '25926' [e2e-llm-inference-service] uid: 47dcf1d8-563a-435e-87ae-27c3dc86164a [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: auth-enabled-test-inference-pool-ip-0e6361b2.e2e-test-llm-auth-enabled-requires-token-d4ed2c08.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: auth-enabled-test-kserve-workload-svc.e2e-test-llm-auth-enabled-requires-token-d4ed2c08.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:32:25Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"3f4fbd10-5888-4617-9e6f-f440212e4c59"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T17:32:25Z' [e2e-llm-inference-service] name: auth-enabled-test-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-auth-enabled-requires-token-d4ed2c08 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: auth-enabled-test [e2e-llm-inference-service] uid: 3f4fbd10-5888-4617-9e6f-f440212e4c59 [e2e-llm-inference-service] resourceVersion: '25686' [e2e-llm-inference-service] uid: bc1093f1-073e-47ea-ad96-f958fe7aabae [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: auth-enabled-test-kserve-workload-svc.e2e-test-llm-auth-enabled-requires-token-d4ed2c08.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: auth-enabled-test-kserve-workload-svc.e2e-test-llm-auth-enabled-requires-token-d4ed2c08.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:32:25Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"3f4fbd10-5888-4617-9e6f-f440212e4c59"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:extensionRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:portNumber: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPortNumber: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T17:32:25Z' [e2e-llm-inference-service] name: auth-enabled-test-inference-pool [e2e-llm-inference-service] namespace: e2e-test-llm-auth-enabled-requires-token-d4ed2c08 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: auth-enabled-test [e2e-llm-inference-service] uid: 3f4fbd10-5888-4617-9e6f-f440212e4c59 [e2e-llm-inference-service] resourceVersion: '25633' [e2e-llm-inference-service] uid: 7ec26383-05c3-44a4-a9bc-e412a91d8eae [e2e-llm-inference-service] spec: [e2e-llm-inference-service] extensionRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: auth-enabled-test-epp-service [e2e-llm-inference-service] portNumber: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPortNumber: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parent: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '1970-01-01T00:00:00Z' [e2e-llm-inference-service] message: Waiting for controller [e2e-llm-inference-service] reason: Pending [e2e-llm-inference-service] status: Unknown [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Status [e2e-llm-inference-service] name: default [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: auth-enabled-test-kserve-router-scheduler-5dbb5856d8-hgkl8 [e2e-llm-inference-service] namespace: e2e-test-llm-auth-enabled-requires-token-d4ed2c08 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:47:17Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 5dbb5856d8 [e2e-llm-inference-service] timestamp: '2026-07-28T17:47:13Z' [e2e-llm-inference-service] window: 15.088s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 545002n [e2e-llm-inference-service] memory: 21808Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_auth.py:380 ⏭️ Skipping deletion of auth-enabled-test due to test failure (SKIP_DELETION_ON_FAILURE=True) [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_auth_enabled_requires_token] [2026-07-28T17:47:17.973152] end - ❌ 902.280s: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T17:32:38Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T17:32:38Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T17:32:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:32:26Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T17:32:38Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T17:32:57Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T17:32:57Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:32:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ---------------------------- Captured log teardown ----------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.conftest:conftest.py:168 Skipping deletion of namespace e2e-test-llm-auth-enabled-requires-token-d4ed2c08 (SKIP_DELETION_ON_FAILURE) [e2e-llm-inference-service] _ test_llm_autoscaling_hpa_deployment[router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-hpa] _ [e2e-llm-inference-service] [gw0] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-hpa'], prom... {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.autoscaling_hpa [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator-no-replicas", [e2e-llm-inference-service] "prometheus-scrape", [e2e-llm-inference-service] "scaling-hpa", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="autoscale-hpa-deploy", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_autoscaling_hpa_deployment(test_case: TestCase): [e2e-llm-inference-service] """HPA + Deployment: HPA exists with WVA annotations; pods scale up under load.""" [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] kserve_client = _new_kserve_client() [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] ns = test_case.namespace [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > _create_and_wait(kserve_client, test_case) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:542: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-hpa'], prom... {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] def _create_and_wait(kserve_client, test_case): [e2e-llm-inference-service] """Create LLMISVC and wait for it to be ready.""" [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:482: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...e-hpa-4c186bcf'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-07-28T17:52:47.261901', start_time = 1785261167.26223 [e2e-llm-inference-service] duration = 900.8762114048004, timestamp_end = '2026-07-28T18:07:48.138444' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....toscale-hpa-4c186bcf'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1376: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7f67b90162a0> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1387: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T17:53:11Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T17:53:11Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T17:53:11Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:52:51Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T17:53:27Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T17:53:27Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T17:53:11Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T17:53:27Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:53:11Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1371: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:82 Created test namespace e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret seaweedfs-s3-creds from kserve-ci-e2e-test to e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret storage-config from kserve-ci-e2e-test to e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:178 Patched default SA in e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 with secret seaweedfs-s3-creds [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:162 ConfigMap odh-kserve-custom-ca-bundle already exists in e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:159 Copied ConfigMap odh-trusted-ca-bundle from kserve-ci-e2e-test to e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig router-managed-autoscale-hpa-de-ec1dce8b in namespace e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig router-managed-autoscale-hpa-de-ec1dce8b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig router-managed-autoscale-hpa-de-ec1dce8b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig workload-llmd-simulator-no-repl-38916baa in namespace e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig workload-llmd-simulator-no-repl-38916baa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig workload-llmd-simulator-no-repl-38916baa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig prometheus-scrape-autoscale-hpa-4c186bcf in namespace e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig prometheus-scrape-autoscale-hpa-4c186bcf [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig prometheus-scrape-autoscale-hpa-4c186bcf [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig scaling-hpa-autoscale-hpa-deplo-347a3180 in namespace e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig scaling-hpa-autoscale-hpa-deplo-347a3180 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig scaling-hpa-autoscale-hpa-deplo-347a3180 [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_autoscaling_hpa_deployment] [2026-07-28T17:52:47.178306] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-hpa'], prompt='KServe is a', service_name='autoscale-hpa-deploy', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, namespace='e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4', before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-deploy', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-de-ec1dce8b'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-38916baa'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-4c186bcf'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-28T17:52:47.191091] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-deploy', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-de-ec1dce8b'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-38916baa'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-4c186bcf'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-28T17:52:47.261800] end - ✅ in 0.070s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-28T17:52:47.261901] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-deploy', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-de-ec1dce8b'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-38916baa'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-4c186bcf'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'RouterReady', 'WorkloadsReady'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T17:52:51Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T17:52:51Z', 'message': 'Inference Pool e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T17:52:51Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:52:51Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T17:52:51Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T17:52:51Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T17:52:51Z', 'message': 'HPA not yet visible in cache', 'reason': 'HPAProgressing', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T17:52:51Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:52:51Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'RouterReady', 'WorkloadsReady'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T17:53:11Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T17:53:11Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T17:53:11Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:52:51Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T17:53:11Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T17:53:11Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T17:53:11Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T17:53:11Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:53:11Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T17:53:11Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T17:53:11Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T17:53:11Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:52:51Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T17:53:27Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T17:53:27Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T17:53:11Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T17:53:27Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:53:11Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 5: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-28T17:29:27Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 6: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-28T17:29:27Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 7: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-28T17:29:27Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1391 Timed out waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T17:53:11Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T17:53:11Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T17:53:11Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:52:51Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T17:53:27Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T17:53:27Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T17:53:11Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T17:53:27Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:53:11Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-07-28T18:07:48.138444] end - ❌ 900.876s: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T17:53:11Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T17:53:11Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T17:53:11Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:52:51Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T17:53:27Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T17:53:27Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T17:53:11Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T17:53:27Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:53:11Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [delete_llmisvc] [2026-07-28T18:07:48.138561] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-deploy', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-de-ec1dce8b'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-38916baa'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-4c186bcf'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 2 pod(s) for autoscale-hpa-deploy still terminating: ['autoscale-hpa-deploy-kserve-66746bd8b9-pbt4s', 'autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-w7bfm'] [e2e-llm-inference-service] assert not ['autoscale-hpa-deploy-kserve-66746bd8b9-pbt4s', 'autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-w7bfm'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 1 pod(s) for autoscale-hpa-deploy still terminating: ['autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-w7bfm'] [e2e-llm-inference-service] assert not ['autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-w7bfm'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [delete_llmisvc] [2026-07-28T18:08:08.416319] end - ✅ in 20.277s [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_autoscaling_hpa_deployment] [2026-07-28T18:08:08.416454] end - ❌ 921.238s: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T17:53:11Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T17:53:11Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T17:53:11Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:52:51Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T17:53:27Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T17:53:27Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T17:53:11Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T17:53:27Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:53:11Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ---------------------------- Captured log teardown ----------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.conftest:conftest.py:168 Skipping deletion of namespace e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 (SKIP_DELETION_ON_FAILURE) [e2e-llm-inference-service] _ test_llm_autoscaling_keda_deployment[router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-keda] _ [e2e-llm-inference-service] [gw0] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-keda'], pro... {'name': 'scaling-keda-autoscale-keda-dep-1ac84077'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.autoscaling_keda [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator-no-replicas", [e2e-llm-inference-service] "prometheus-scrape", [e2e-llm-inference-service] "scaling-keda", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="autoscale-keda-deploy", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_autoscaling_keda_deployment(test_case: TestCase): [e2e-llm-inference-service] """KEDA + Deployment: ScaledObject exists with WVA annotations; no HPA; pods scale up under load.""" [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] kserve_client = _new_kserve_client() [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] ns = test_case.namespace [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > _create_and_wait(kserve_client, test_case) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:606: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-keda'], pro... {'name': 'scaling-keda-autoscale-keda-dep-1ac84077'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] def _create_and_wait(kserve_client, test_case): [e2e-llm-inference-service] """Create LLMISVC and wait for it to be ready.""" [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:482: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...e-ked-101f2a9d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-dep-1ac84077'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-07-28T18:08:09.737108', start_time = 1785262089.7373936 [e2e-llm-inference-service] duration = 900.6353406906128, timestamp_end = '2026-07-28T18:23:10.372746' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....toscale-ked-101f2a9d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-dep-1ac84077'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1376: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7f67b9c8cae0> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1387: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T18:08:43Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:08:43Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:08:43Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:08:27Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:08:58Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:08:58Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:08:43Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:08:58Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:08:43Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1371: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:82 Created test namespace e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret seaweedfs-s3-creds from kserve-ci-e2e-test to e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret storage-config from kserve-ci-e2e-test to e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:178 Patched default SA in e2e-test-llm-autoscaling-keda-deployment-b2150d0b with secret seaweedfs-s3-creds [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:162 ConfigMap odh-kserve-custom-ca-bundle already exists in e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:159 Copied ConfigMap odh-trusted-ca-bundle from kserve-ci-e2e-test to e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig router-managed-autoscale-keda-d-27e06c40 in namespace e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig router-managed-autoscale-keda-d-27e06c40 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig router-managed-autoscale-keda-d-27e06c40 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig workload-llmd-simulator-no-repl-da49e827 in namespace e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig workload-llmd-simulator-no-repl-da49e827 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig workload-llmd-simulator-no-repl-da49e827 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig prometheus-scrape-autoscale-ked-101f2a9d in namespace e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig prometheus-scrape-autoscale-ked-101f2a9d [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig prometheus-scrape-autoscale-ked-101f2a9d [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig scaling-keda-autoscale-keda-dep-1ac84077 in namespace e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig scaling-keda-autoscale-keda-dep-1ac84077 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig scaling-keda-autoscale-keda-dep-1ac84077 [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_autoscaling_keda_deployment] [2026-07-28T18:08:09.570736] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-keda'], prompt='KServe is a', service_name='autoscale-keda-deploy', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, namespace='e2e-test-llm-autoscaling-keda-deployment-b2150d0b', before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-keda-deploy', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-keda-deployment-b2150d0b', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-keda-d-27e06c40'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-da49e827'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-ked-101f2a9d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-dep-1ac84077'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-28T18:08:09.583084] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-keda-deploy', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-keda-deployment-b2150d0b', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-keda-d-27e06c40'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-da49e827'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-ked-101f2a9d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-dep-1ac84077'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-28T18:08:09.736885] end - ✅ in 0.154s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-28T18:08:09.737108] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-keda-deploy', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-keda-deployment-b2150d0b', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-keda-d-27e06c40'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-da49e827'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-ked-101f2a9d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-dep-1ac84077'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'RouterReady', 'WorkloadsReady'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T18:08:27Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:08:27Z', 'message': 'Inference Pool e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:08:27Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:08:27Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:08:27Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:08:27Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:08:27Z', 'message': 'ScaledObject not yet visible in cache', 'reason': 'ScaledObjectProgressing', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:08:27Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:08:27Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'RouterReady', 'WorkloadsReady'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T18:08:43Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:08:43Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:08:43Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:08:27Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:08:43Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:08:43Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:08:43Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:08:43Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:08:43Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T18:08:43Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:08:43Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:08:43Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:08:27Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:08:58Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:08:58Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:08:43Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:08:58Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:08:43Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 8: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-28T17:29:27Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 9: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-28T17:29:27Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 10: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-28T17:29:27Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1391 Timed out waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T18:08:43Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:08:43Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:08:43Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:08:27Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:08:58Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:08:58Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:08:43Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:08:58Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:08:43Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-07-28T18:23:10.372746] end - ❌ 900.635s: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T18:08:43Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:08:43Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:08:43Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:08:27Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:08:58Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:08:58Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:08:43Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:08:58Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:08:43Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [delete_llmisvc] [2026-07-28T18:23:10.372853] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-keda-deploy', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-keda-deployment-b2150d0b', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-keda-d-27e06c40'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-da49e827'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-ked-101f2a9d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-dep-1ac84077'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 2 pod(s) for autoscale-keda-deploy still terminating: ['autoscale-keda-deploy-kserve-77fc6d7cf9-ccq5g', 'autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-mts9k'] [e2e-llm-inference-service] assert not ['autoscale-keda-deploy-kserve-77fc6d7cf9-ccq5g', 'autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-mts9k'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 1 pod(s) for autoscale-keda-deploy still terminating: ['autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-mts9k'] [e2e-llm-inference-service] assert not ['autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-mts9k'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [delete_llmisvc] [2026-07-28T18:23:45.576493] end - ✅ in 35.203s [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_autoscaling_keda_deployment] [2026-07-28T18:23:45.576719] end - ❌ 936.006s: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T18:08:43Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:08:43Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:08:43Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:08:27Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:08:58Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:08:58Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:08:43Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:08:58Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:08:43Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ---------------------------- Captured log teardown ----------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.conftest:conftest.py:168 Skipping deletion of namespace e2e-test-llm-autoscaling-keda-deployment-b2150d0b (SKIP_DELETION_ON_FAILURE) [e2e-llm-inference-service] _____ test_llm_inference_service[router-managed-workload-pd-cpu-model-pvc] _____ [e2e-llm-inference-service] [gw1] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-pd-cpu', 'model-pvc'], prompt='KServe is a', service_name='llmisvc-mod... {'name': 'model-pvc-llmisvc-model-pvc-rou-49c1f027'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-gateway-ref", [e2e-llm-inference-service] "router-with-managed-route", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] expected_gateway="router-gateway-1", [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda tc: create_router_resources( [e2e-llm-inference-service] gateways=[ [e2e-llm-inference-service] make_router_gateway( [e2e-llm-inference-service] "router-gateway-1", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="custom-route-timeout-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="router-with-refs-test", [e2e-llm-inference-service] expected_gateway="router-gateway-1", [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda tc: create_router_resources( [e2e-llm-inference-service] gateways=[ [e2e-llm-inference-service] make_router_gateway( [e2e-llm-inference-service] "router-gateway-1", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] routes=[ [e2e-llm-inference-service] make_router_main_route( [e2e-llm-inference-service] "router-route-1", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] "router-gateway-1", [e2e-llm-inference-service] "router-with-refs-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] make_router_health_route( [e2e-llm-inference-service] "router-route-2", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] "router-gateway-1", [e2e-llm-inference-service] "router-with-refs-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=["router-managed", "workload-pd-cpu", "model-fb-opt-125m"], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="custom-route-timeout-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="router-with-refs-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] expected_gateway="router-gateway-2", [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda tc: create_router_resources( [e2e-llm-inference-service] gateways=[ [e2e-llm-inference-service] make_router_gateway( [e2e-llm-inference-service] "router-gateway-2", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] routes=[ [e2e-llm-inference-service] make_router_main_route( [e2e-llm-inference-service] "router-route-3", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] "router-gateway-2", [e2e-llm-inference-service] "router-with-refs-pd-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] make_router_health_route( [e2e-llm-inference-service] "router-route-4", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] "router-gateway-2", [e2e-llm-inference-service] "router-with-refs-pd-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-dp-ep-gpu", [e2e-llm-inference-service] "workload-dp-ep-prefill-gpu", [e2e-llm-inference-service] "model-deepseek-v2-lite", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="Delve into the multifaceted implications of a fully disaggregated cloud architecture, specifically " [e2e-llm-inference-service] "where the compute plane (P) and the data plane (D) are independently deployed and managed for a " [e2e-llm-inference-service] "geographically distributed, high-throughput, low-latency microservices ecosystem. Beyond the " [e2e-llm-inference-service] "fundamental challenges of network latency and data consistency, elaborate on the advanced " [e2e-llm-inference-service] "considerations and trade-offs inherent in such a setup: 1. Network Architecture and Protocols: " [e2e-llm-inference-service] "How would the network fabric and underlying protocols (e.g., RDMA, custom transport layers) need to " [e2e-llm-inference-service] "evolve to support optimal performance and minimize inter-plane communication overhead, especially for " [e2e-llm-inference-service] "synchronous operations? Discuss the role of network programmability (e.g., SDN, P4) in dynamically " [e2e-llm-inference-service] "optimizing routing and traffic flow between P and D. 2. Advanced Data Consistency and Durability: " [e2e-llm-inference-service] "Explore sophisticated data consistency models (e.g., causal consistency, strong eventual consistency) " [e2e-llm-inference-service] "and their applicability in balancing performance and data integrity across a globally distributed data plane. " [e2e-llm-inference-service] "Detail strategies for ensuring data durability and fault tolerance, including multi-region replication, " [e2e-llm-inference-service] "intelligent partitioning, and recovery mechanisms in the event of partial or full plane failures. " [e2e-llm-inference-service] "3. Dynamic Resource Orchestration and Cost Optimization: Analyze how an orchestration layer would intelligently " [e2e-llm-inference-service] "manage the independent scaling of compute (P) and data (D) resources, considering fluctuating workloads, " [e2e-llm-inference-service] "cost efficiency, and performance targets (e.g., using predictive analytics for resource provisioning). " [e2e-llm-inference-service] "Discuss mechanisms for dynamically reallocating compute nodes to different data partitions based on " [e2e-llm-inference-service] "workload patterns and data locality, potentially involving live migration strategies. " [e2e-llm-inference-service] "4. Security and Compliance in a Distributed Landscape: Address the enhanced security perimeter " [e2e-llm-inference-service] "challenges, including securing communication channels between P and D (encryption in transit, mutual TLS), " [e2e-llm-inference-service] "fine-grained access control to data at rest and in motion, and identity management across disaggregated " [e2e-llm-inference-service] "components. Discuss how such an architecture impacts compliance with regulatory frameworks (e.g., GDPR, HIPAA) " [e2e-llm-inference-service] "concerning data sovereignty, privacy, and auditability. 5. Operational Complexity and Observability: " [e2e-llm-inference-service] "Examine the increased complexity in monitoring, logging, and tracing across highly decoupled compute and " [e2e-llm-inference-service] "data planes. What specialized tooling and practices (e.g., distributed tracing with OpenTelemetry, advanced AIOps) " [e2e-llm-inference-service] "would be essential? How would incident response and troubleshooting differ in this disaggregated environment " [e2e-llm-inference-service] "compared to traditional integrated systems? Consider the challenges of pinpointing root causes across " [e2e-llm-inference-service] "independent failures. 6. Real-world Applicability and Future Trends: Identify specific industries " [e2e-llm-inference-service] "or use cases (e.g., high-frequency trading, IoT edge processing, large language model inference) " [e2e-llm-inference-service] "where the benefits of P/D disaggregation would strongly outweigh its complexities. " [e2e-llm-inference-service] "Conclude by speculating on emerging technologies or paradigms (e.g., serverless compute functions " [e2e-llm-inference-service] "directly interacting with object storage, in-memory disaggregation) that could further drive or " [e2e-llm-inference-service] "transform P/D disaggregation in cloud computing.", [e2e-llm-inference-service] max_tokens=2000, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_gpu, [e2e-llm-inference-service] pytest.mark.cluster_nvidia, [e2e-llm-inference-service] pytest.mark.cluster_nvidia_roce, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-no-scheduler", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.no_scheduler, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="This test simulates DP+EP that can run on CPU, the idea is to test the LWS-based deployment, " [e2e-llm-inference-service] "but without the resources requirements for DP+EP (GPUs and ROCe/IB).", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_multi_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler config tests [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-inline-config-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Chat completions endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] model_name="Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-configmap-ref", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-configmap-ref-test", [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda tc: create_scheduler_configmap(namespace=tc.namespace) [e2e-llm-inference-service] ], [e2e-llm-inference-service] after_test=[ [e2e-llm-inference-service] lambda tc: delete_scheduler_configmap(namespace=tc.namespace) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-replicas", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-ha-replicas-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-custom-template", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-custom-template-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler v0.6 → v0.7 migration tests. [e2e-llm-inference-service] # Deploy v0.6-style configs and verify the controller migrates them [e2e-llm-inference-service] # so the v0.7 scheduler boots successfully. [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-v06-pd-config-migration", [e2e-llm-inference-service] "workload-llmd-simulator-pd", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-v06-pd-migration-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-v06-nonzero-threshold-migration", [e2e-llm-inference-service] "workload-llmd-simulator-pd", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-v06-threshold-migration-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Standalone tokenizer — clean path: token-producer in inline config [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-tokenizer-kvcache", [e2e-llm-inference-service] "workload-llmd-simulator-kvcache", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="tokenizer-clean-path-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Standalone tokenizer — migration path: legacy precise-prefix-cache-scorer [e2e-llm-inference-service] # triggers auto-provisioned tokenizer without explicit tokenizer:{} field [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-precise-prefix-cache-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator-kvcache", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="tokenizer-migration-path-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Models endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="data"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/chat/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — LoRA adapter [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] model_name="publishers/{namespace}/models/lora-adapter-1", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "publishers/{namespace}/models/lora-adapter-1" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/lora-adapter-1", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/models (base + LoRA) [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=assert_models_contains( [e2e-llm-inference-service] "facebook/opt-125m", [e2e-llm-inference-service] "publishers/{namespace}/models/facebook/opt-125m", [e2e-llm-inference-service] "lora-adapter-1", [e2e-llm-inference-service] "publishers/{namespace}/models/lora-adapter-1", [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # PVC storage tests -- validate direct PVC volume mount with real vLLM serving [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] before_test=[lambda tc: ensure_pvc_with_model(namespace=tc.namespace)], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] before_test=[lambda tc: ensure_pvc_with_model(namespace=tc.namespace)], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] before_test=[lambda tc: ensure_pvc_with_model(namespace=tc.namespace)], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_multi_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_inference_service(test_case: TestCase): # noqa: F811 [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = KServeClient( [e2e-llm-inference-service] config_file=os.environ.get("KUBECONFIG", "~/.kube/config"), [e2e-llm-inference-service] client_configuration=client.Configuration(), [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] prefix = test_case.log_prefix [e2e-llm-inference-service] [e2e-llm-inference-service] test_failed = False [e2e-llm-inference-service] try: [e2e-llm-inference-service] print(f"{prefix} Creating LLMInferenceService {service_name}") [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] print(f"{prefix} Waiting for LLMInferenceService {service_name} to be ready") [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:866: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...del-p-9d807ba3'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-49c1f027'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-07-28T18:12:47.647291', start_time = 1785262367.6475894 [e2e-llm-inference-service] duration = 900.5906484127045, timestamp_end = '2026-07-28T18:27:48.238240' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....svc-model-p-9d807ba3'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-49c1f027'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1376: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7ff1238d9b20> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1387: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T18:13:12Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:13:12Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:13:12Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:15:46Z', 'severity': 'Info', 'status': 'True', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:13:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:13:12Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:13:36Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:13:36Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:13:12Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1371: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:82 Created test namespace e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret seaweedfs-s3-creds from kserve-ci-e2e-test to e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret storage-config from kserve-ci-e2e-test to e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:178 Patched default SA in e2e-test-llm-inference-service-48639af5 with secret seaweedfs-s3-creds [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:162 ConfigMap odh-kserve-custom-ca-bundle already exists in e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:159 Copied ConfigMap odh-trusted-ca-bundle from kserve-ci-e2e-test to e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1959 Created PVC e2e-pvc-model-storage in namespace e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2099 Created model download Job e2e-pvc-model-download in namespace e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2124 Model download Job e2e-pvc-model-download completed successfully [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig router-managed-llmisvc-model-pv-d968e7b0 in namespace e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig router-managed-llmisvc-model-pv-d968e7b0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig router-managed-llmisvc-model-pv-d968e7b0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig workload-pd-cpu-llmisvc-model-p-9d807ba3 in namespace e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig workload-pd-cpu-llmisvc-model-p-9d807ba3 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig workload-pd-cpu-llmisvc-model-p-9d807ba3 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig model-pvc-llmisvc-model-pvc-rou-49c1f027 in namespace e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig model-pvc-llmisvc-model-pvc-rou-49c1f027 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig model-pvc-llmisvc-model-pvc-rou-49c1f027 [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_inference_service] [2026-07-28T18:12:47.489097] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-pd-cpu', 'model-pvc'], prompt='KServe is a', service_name='llmisvc-model-pvc-router-manage-e8706282', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, namespace='e2e-test-llm-inference-service-48639af5', before_test=[ at 0x7ff1283237e0>], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-pvc-router-manage-e8706282', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-inference-service-48639af5', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-pv-d968e7b0'}, [e2e-llm-inference-service] {'name': 'workload-pd-cpu-llmisvc-model-p-9d807ba3'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-49c1f027'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-28T18:12:47.501377] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-pvc-router-manage-e8706282', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-inference-service-48639af5', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-pv-d968e7b0'}, [e2e-llm-inference-service] {'name': 'workload-pd-cpu-llmisvc-model-p-9d807ba3'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-49c1f027'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-28T18:12:47.647116] end - ✅ in 0.146s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-28T18:12:47.647291] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-pvc-router-manage-e8706282', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-inference-service-48639af5', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-pv-d968e7b0'}, [e2e-llm-inference-service] {'name': 'workload-pd-cpu-llmisvc-model-p-9d807ba3'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-49c1f027'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T18:13:03Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:13:03Z', 'message': 'Inference Pool e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:13:03Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:13:03Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:13:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:13:03Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:13:03Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:13:03Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:13:03Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T18:13:12Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:13:12Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:13:12Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:13:12Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:13:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:13:12Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:13:12Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:13:12Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:13:12Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T18:13:12Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:13:12Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:13:12Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:13:12Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:13:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:13:12Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:13:36Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:13:36Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:13:12Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T18:13:12Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:13:12Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:13:12Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:15:46Z', 'severity': 'Info', 'status': 'True', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:13:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:13:12Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:13:36Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:13:36Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:13:12Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1391 Timed out waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T18:13:12Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:13:12Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:13:12Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:15:46Z', 'severity': 'Info', 'status': 'True', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:13:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:13:12Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:13:36Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:13:36Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:13:12Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-07-28T18:27:48.238240] end - ❌ 900.591s: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T18:13:12Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:13:12Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:13:12Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:15:46Z', 'severity': 'Info', 'status': 'True', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:13:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:13:12Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:13:36Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:13:36Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:13:12Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:903 [router-managed-workload-pd-cpu-model-pvc] ❌ ERROR: Failed to call llm inference service llmisvc-model-pvc-router-manage-e8706282: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T18:13:12Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:13:12Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:13:12Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:15:46Z', 'severity': 'Info', 'status': 'True', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:13:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:13:12Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:13:36Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:13:36Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:13:12Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:243 # Diagnostics for 'llmisvc-model-pvc-router-manage-e8706282' in 'e2e-test-llm-inference-service-48639af5' [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:244 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:247 # LLMInferenceService llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:256 apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] security.opendatahub.io/enable-auth: 'false' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:12:47Z' [e2e-llm-inference-service] finalizers: [e2e-llm-inference-service] - serving.kserve.io/llmisvc-finalizer [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:security.opendatahub.io/enable-auth: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:baseRefs: {} [e2e-llm-inference-service] manager: OpenAPI-Generator [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T18:12:47Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:finalizers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] v:"serving.kserve.io/llmisvc-finalizer": {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T18:12:47Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:addresses: {} [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-router-route: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-scheduler: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-scheduler-latency-predictor: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-tokenizer: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-tracing: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-worker-data-parallel: {} [e2e-llm-inference-service] f:appliedConfigs: {} [e2e-llm-inference-service] f:conditions: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:router: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:gateways: {} [e2e-llm-inference-service] f:scheduler: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:inferencePool: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:service: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:url: {} [e2e-llm-inference-service] f:workloads: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:prefill: {} [e2e-llm-inference-service] f:primary: {} [e2e-llm-inference-service] f:scheduler: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-28T18:15:46Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] resourceVersion: '65430' [e2e-llm-inference-service] uid: c0bf7073-efea-42d3-83eb-e33d2c9137c1 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] baseRefs: [e2e-llm-inference-service] - name: router-managed-llmisvc-model-pv-d968e7b0 [e2e-llm-inference-service] - name: workload-pd-cpu-llmisvc-model-p-9d807ba3 [e2e-llm-inference-service] - name: model-pvc-llmisvc-model-pvc-rou-49c1f027 [e2e-llm-inference-service] model: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uri: '' [e2e-llm-inference-service] status: [e2e-llm-inference-service] addresses: [e2e-llm-inference-service] - name: gateway-external-model-routing [e2e-llm-inference-service] url: http://a803f1ae9e25d4fc48c1c4d8072319d1-1138438229.us-east-1.elb.amazonaws.com/ [e2e-llm-inference-service] - name: gateway-external [e2e-llm-inference-service] url: http://a803f1ae9e25d4fc48c1c4d8072319d1-1138438229.us-east-1.elb.amazonaws.com/e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] - name: gateway-external [e2e-llm-inference-service] url: http://a803f1ae9e25d4fc48c1c4d8072319d1-1138438229.us-east-1.elb.amazonaws.com/publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] - name: gateway-internal-model-routing [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/ [e2e-llm-inference-service] - name: gateway-internal [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] - name: gateway-internal [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-template: kserve-config-llm-decode-template [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-worker-data-parallel: kserve-config-llm-decode-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-template: kserve-config-llm-prefill-template [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-worker-data-parallel: kserve-config-llm-prefill-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-router-route: kserve-config-llm-router-route [e2e-llm-inference-service] serving.kserve.io/config-llm-scheduler: kserve-config-llm-scheduler [e2e-llm-inference-service] serving.kserve.io/config-llm-scheduler-latency-predictor: kserve-config-llm-scheduler-latency-predictor [e2e-llm-inference-service] serving.kserve.io/config-llm-template: kserve-config-llm-template [e2e-llm-inference-service] serving.kserve.io/config-llm-tokenizer: kserve-config-llm-tokenizer [e2e-llm-inference-service] serving.kserve.io/config-llm-tracing: kserve-config-llm-tracing [e2e-llm-inference-service] serving.kserve.io/config-llm-worker-data-parallel: kserve-config-llm-worker-data-parallel [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T18:13:12Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: HTTPRoutesReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T18:13:12Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: InferencePoolReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T18:13:12Z' [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: MainWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T18:15:46Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: PrefillWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T18:13:03Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: PresetsCombined [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T18:13:12Z' [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: Ready [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T18:13:36Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: RouterReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T18:13:36Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: SchedulerWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T18:13:12Z' [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: WorkloadsReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] url: http://a803f1ae9e25d4fc48c1c4d8072319d1-1138438229.us-east-1.elb.amazonaws.com/e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:68 TIME NAMESPACE SOURCE TYPE REASON MESSAGE [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:69 -------------------------------------------------------------------------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None e2e-test-llm-inference-service-48639af5 Normal Scheduled Successfully assigned e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-s2mpj to ip-10-0-135-20.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:12:32 e2e-test-llm-inference-service-48639af5 attachdetach-controller Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-f1df9520-ed9a-482b-9b2b-0f8681809f16" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:12:35 e2e-test-llm-inference-service-48639af5 multus Normal AddedInterface Add eth0 [10.134.0.47/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:12:35 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-135-20.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:12:35 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-135-20.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:12:35 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-135-20.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:12:27 e2e-test-llm-inference-service-48639af5 job-controller Normal SuccessfulCreate Created pod: e2e-pvc-model-download-s2mpj [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:12:46 e2e-test-llm-inference-service-48639af5 job-controller Normal Completed Job completed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:12:27 e2e-test-llm-inference-service-48639af5 persistentvolume-controller Normal WaitForFirstConsumer waiting for first consumer to be created before binding [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:12:27 e2e-test-llm-inference-service-48639af5 ebs.csi.aws.com_aws-ebs-csi-driver-controller-6d6bb9d567-5nkw8_6545112e-a05c-4b1c-bbed-7303f5f341a7 Normal Provisioning External provisioner is provisioning volume for claim "e2e-test-llm-inference-service-48639af5/e2e-pvc-model-storage" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:12:28 e2e-test-llm-inference-service-48639af5 persistentvolume-controller Normal ExternalProvisioning Waiting for a volume to be created either by the external provisioner 'ebs.csi.aws.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:12:29 e2e-test-llm-inference-service-48639af5 ebs.csi.aws.com_aws-ebs-csi-driver-controller-6d6bb9d567-5nkw8_6545112e-a05c-4b1c-bbed-7303f5f341a7 Normal ProvisioningSucceeded Successfully provisioned volume pvc-f1df9520-ed9a-482b-9b2b-0f8681809f16 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:13:01 e2e-test-llm-inference-service-48639af5 replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None e2e-test-llm-inference-service-48639af5 Normal Scheduled Successfully assigned e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs to ip-10-0-135-20.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:13:05 e2e-test-llm-inference-service-48639af5 attachdetach-controller Warning FailedAttachVolume Multi-Attach error for volume "pvc-f1df9520-ed9a-482b-9b2b-0f8681809f16" Volume is already used by pod(s) llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:13:01 e2e-test-llm-inference-service-48639af5 replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None e2e-test-llm-inference-service-48639af5 Normal Scheduled Successfully assigned e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd to ip-10-0-135-188.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:13:05 e2e-test-llm-inference-service-48639af5 attachdetach-controller Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-f1df9520-ed9a-482b-9b2b-0f8681809f16" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:13:06 e2e-test-llm-inference-service-48639af5 multus Normal AddedInterface Add eth0 [10.132.0.61/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:13:06 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-135-188.ec2.internal Normal Pulling Pulling image "vllm/vllm-openai-cpu:v0.19.0" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:13:37 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-135-188.ec2.internal Normal Pulled Successfully pulled image "vllm/vllm-openai-cpu:v0.19.0" in 30.765s (30.765s including waiting). Image size: 3531178100 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:13:37 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-135-188.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:13:37 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-135-188.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:15:36 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-135-188.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.61:8000/health": dial tcp 10.132.0.61:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:13:01 e2e-test-llm-inference-service-48639af5 deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b89cd7957 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:13:01 e2e-test-llm-inference-service-48639af5 deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:12:57 e2e-test-llm-inference-service-48639af5 OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-pvc-router-manage-e8706282-kserve-route-authn: failed to get HTTPRoute e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-pvc-router-manage-e8706282-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:13:01 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.Secret e2e-test-llm-inference-service-48639af5/llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:13:01 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.ServiceAccount e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:13:01 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.Role e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:13:01 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.RoleBinding e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:13:01 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.Deployment e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:13:01 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.Deployment e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:13:01 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.Service e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:13:01 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.ServiceAccount e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:13:01 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /e2e-test-llm-inference-service-132820f03254470cd8a0146008974a5f [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:13:10 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:23:09 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Updated Updated v1.Secret e2e-test-llm-inference-service-48639af5/llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:13:10 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None e2e-test-llm-inference-service-48639af5 Normal Scheduled Successfully assigned e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche6mt66 to ip-10-0-135-20.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:13:01 e2e-test-llm-inference-service-48639af5 multus Normal AddedInterface Add eth0 [10.134.0.49/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:13:02 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-135-20.ec2.internal Normal Pulled Container image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:13:02 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-135-20.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:13:02 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-135-20.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:13:01 e2e-test-llm-inference-service-48639af5 replicaset-controller Normal SuccessfulCreate Created pod: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche6mt66 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:13:01 e2e-test-llm-inference-service-48639af5 deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-7fc69df7b8 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:172 ### Pod llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs (phase=Pending) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:212 #### init-container 'llm-d-routing-sidecar' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:228 # -- logs (current): unavailable ((400) [e2e-llm-inference-service] Reason: Bad Request [e2e-llm-inference-service] HTTP response headers: HTTPHeaderDict({'Audit-Id': '28dac29b-f166-4c41-9ea5-34ef635a3f4f', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'Date': 'Tue, 28 Jul 2026 18:27:48 GMT', 'Content-Length': '262'}) [e2e-llm-inference-service] HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"container \"llm-d-routing-sidecar\" in pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs\" is waiting to start: PodInitializing","reason":"BadRequest","code":400} [e2e-llm-inference-service] [e2e-llm-inference-service] ) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:212 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:228 # -- logs (current): unavailable ((400) [e2e-llm-inference-service] Reason: Bad Request [e2e-llm-inference-service] HTTP response headers: HTTPHeaderDict({'Audit-Id': '1f483d32-b71e-42d7-a636-03b0932eec0b', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'Date': 'Tue, 28 Jul 2026 18:27:48 GMT', 'Content-Length': '245'}) [e2e-llm-inference-service] HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"container \"main\" in pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs\" is waiting to start: PodInitializing","reason":"BadRequest","code":400} [e2e-llm-inference-service] [e2e-llm-inference-service] ) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:172 ### Pod llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:212 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:225 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:226 (APIServer pid=1) INFO 07-28 18:15:37 [entrypoints/launcher.py:46] Route: /health, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 18:15:37 [entrypoints/launcher.py:46] Route: /metrics, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 18:15:37 [entrypoints/launcher.py:46] Route: /v1/models, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 18:15:37 [entrypoints/launcher.py:46] Route: /ping, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 18:15:37 [entrypoints/launcher.py:46] Route: /ping, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 18:15:37 [entrypoints/launcher.py:46] Route: /invocations, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 18:15:37 [entrypoints/launcher.py:46] Route: /v1/chat/completions, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 18:15:37 [entrypoints/launcher.py:46] Route: /v1/chat/completions/batch, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 18:15:37 [entrypoints/launcher.py:46] Route: /v1/responses, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 18:15:37 [entrypoints/launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 18:15:37 [entrypoints/launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 18:15:37 [entrypoints/launcher.py:46] Route: /v1/completions, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 18:15:37 [entrypoints/launcher.py:46] Route: /v1/messages, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 18:15:37 [entrypoints/launcher.py:46] Route: /v1/messages/count_tokens, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 18:15:37 [entrypoints/launcher.py:46] Route: /inference/v1/generate, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 18:15:37 [entrypoints/launcher.py:46] Route: /scale_elastic_ep, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 18:15:37 [entrypoints/launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 18:15:37 [entrypoints/launcher.py:46] Route: /v1/chat/completions/render, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 18:15:37 [entrypoints/launcher.py:46] Route: /v1/completions/render, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO: Started server process [1] [e2e-llm-inference-service] (APIServer pid=1) INFO: Waiting for application startup. [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 18:15:37 [entrypoints/ssl.py:60] SSLCertRefresher monitors files: ['/var/run/kserve/tls/tls.key', '/var/run/kserve/tls/tls.crt'] [e2e-llm-inference-service] (APIServer pid=1) INFO: Application startup complete. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:15:46 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:15:46 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 18:15:46 [entrypoints/ssl.py:64] File change detected: modified - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 18:15:46 [entrypoints/ssl.py:34] Reloading SSL certificate chain [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 18:15:46 [entrypoints/ssl.py:64] File change detected: modified - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 18:15:46 [entrypoints/ssl.py:34] Reloading SSL certificate chain [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 18:15:46 [entrypoints/ssl.py:64] File change detected: deleted - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 18:15:46 [entrypoints/ssl.py:34] Reloading SSL certificate chain [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 18:15:46 [entrypoints/ssl.py:64] File change detected: deleted - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 18:15:46 [entrypoints/ssl.py:34] Reloading SSL certificate chain [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:15:48 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:15:56 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:15:58 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:16:06 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:16:08 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:16:16 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:16:18 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:16:26 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:16:28 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:16:36 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:16:38 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:16:46 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:16:48 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:16:56 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:16:58 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:17:06 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:17:06 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:17:08 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:17:16 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:17:18 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:17:26 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:17:28 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:17:36 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:17:36 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:17:38 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:17:46 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:17:48 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:17:56 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:17:58 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:18:06 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:18:06 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:18:08 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:18:16 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:18:18 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:18:26 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:18:28 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:18:36 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:18:36 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:18:38 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:18:46 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:18:48 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:18:56 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:18:58 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:19:06 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:19:06 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:19:08 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:19:16 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:19:18 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:19:26 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:19:28 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:19:36 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:19:36 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:19:38 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:19:46 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:19:48 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:19:56 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:19:58 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:20:06 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:20:06 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:20:08 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:20:16 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:20:18 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:20:26 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:20:28 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:20:36 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:20:36 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:20:38 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:20:46 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:20:48 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:20:56 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:20:58 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:21:06 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:21:06 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:21:08 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:21:16 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:21:18 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:21:26 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:21:28 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:21:36 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:21:36 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:21:38 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:21:46 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:21:48 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:21:56 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:21:58 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:22:06 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:22:06 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:22:08 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:22:16 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:22:18 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:22:26 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:22:28 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:22:36 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:22:36 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:22:38 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:22:46 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:22:48 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:22:56 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:22:58 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:23:06 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:23:06 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:23:08 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:23:16 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:23:18 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:23:26 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:23:28 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:23:36 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:23:36 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:23:38 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:23:46 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:23:48 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:23:56 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:23:58 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:24:06 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:24:06 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:24:08 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:24:16 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:24:18 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:24:26 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:24:28 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:24:36 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:24:36 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:24:38 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:24:46 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:24:48 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:24:56 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:24:58 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:25:06 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:25:06 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:25:08 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:25:16 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:25:18 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:25:26 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:25:28 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:25:36 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:25:36 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:25:38 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:25:46 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:25:48 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:25:56 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:25:58 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:26:06 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:26:06 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:26:08 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:26:16 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:26:18 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:26:26 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:26:28 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:26:36 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:26:36 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:26:38 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:26:46 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:26:48 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:26:56 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:26:58 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:27:06 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:27:06 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:27:08 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:27:16 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:27:18 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:27:26 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:27:28 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:27:36 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:27:36 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:27:38 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:27:46 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:27:48 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:172 ### Pod llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche6mt66 (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:212 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:225 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:226 {"level":"info","ts":1785262382.0998693,"logger":"setup","caller":"runner/runner.go:196","msg":"GIE build","commit-sha":"unknown","build-ref":""} [e2e-llm-inference-service] {"level":"info","ts":1785262382.0999663,"logger":"setup","caller":"runner/runner.go:217","msg":"Flags processed","flags":{"cert-path":"/var/run/kserve/tls","config-file":"","config-text":"apiVersion: llm-d.ai/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n- type: disagg-headers-handler\n- type: prefill-filter\n- type: decode-filter\n- type: queue-scorer\n- type: kv-cache-utilization-scorer\n- type: active-request-scorer\n- type: prefix-cache-scorer\n- type: max-score-picker\n- type: always-disagg-pd-decider\n- parameters:\n deciders:\n prefill: always-disagg-pd-decider\n type: disagg-profile-handler\n- parameters:\n scheme: https\n type: metrics-data-source\nschedulingProfiles:\n- name: prefill\n plugins:\n - pluginRef: prefill-filter\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: queue-scorer\n weight: 2\n - pluginRef: kv-cache-utilization-scorer\n weight: 2\n - pluginRef: max-score-picker\n- name: decode\n plugins:\n - pluginRef: decode-filter\n - pluginRef: active-request-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n","disable-endpoint-subset-filter":false,"enable-cert-reload":true,"enable-grpc-stream-metrics":false,"enable-pprof":true,"endpoint-selector":"","endpoint-target-ports":{},"grpc-health-port":9003,"grpc-max-recv-msg-size":"","grpc-max-send-msg-size":"","grpc-port":9002,"ha-enable-leader-election":false,"health-checking":false,"metrics-endpoint-auth":true,"metrics-port":9090,"metrics-staleness-threshold":2000000000,"model-server-metrics-https-insecure-skip-verify":true,"model-server-metrics-path":"/metrics","model-server-metrics-port":0,"model-server-metrics-scheme":"http","pool-group":"inference.networking.k8s.io","pool-name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool","pool-namespace":"e2e-test-llm-inference-service-48639af5","refresh-metrics-interval":50000000,"refresh-prometheus-metrics-interval":5000000000,"secure-serving":true,"tracing":true,"v":2,"zap-devel":{},"zap-encoder":{},"zap-log-level":{},"zap-stacktrace-level":{},"zap-time-encoding":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785262382.1000943,"logger":"setup.trace","caller":"tracing/telemetry.go:123","msg":"init OTel trace exporter","type":"console"} [e2e-llm-inference-service] {"level":"info","ts":1785262382.1015427,"caller":"loader/configloader.go:121","msg":"Loaded raw configuration","config":"{Plugins: [{Type: disagg-headers-handler} {Type: prefill-filter} {Type: decode-filter} {Type: queue-scorer} {Type: kv-cache-utilization-scorer} {Type: active-request-scorer} {Type: prefix-cache-scorer} {Type: max-score-picker} {Type: always-disagg-pd-decider} {Type: disagg-profile-handler, Parameters: {\"deciders\":{\"prefill\":\"always-disagg-pd-decider\"}}} {Type: metrics-data-source, Parameters: {\"scheme\":\"https\"}}], SchedulingProfiles: [{Name: prefill, Plugins: [{PluginRef: prefill-filter} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: queue-scorer, Weight: 2.00} {PluginRef: kv-cache-utilization-scorer, Weight: 2.00} {PluginRef: max-score-picker}]} {Name: decode, Plugins: [{PluginRef: decode-filter} {PluginRef: active-request-scorer, Weight: 2.00} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: max-score-picker}]}]}"} [e2e-llm-inference-service] {"level":"info","ts":1785262382.101619,"logger":"setup","caller":"runner/runner.go:622","msg":"Data layer: ENABLED"} [e2e-llm-inference-service] {"level":"info","ts":1785262382.1019318,"logger":"setup","caller":"runner/runner.go:281","msg":"Raw config after phase one","config":{"apiVersion":"llm-d.ai/v1alpha1","dataLayer":null,"kind":"EndpointPickerConfig","plugins":[{"name":"disagg-headers-handler","parameters":null,"type":"disagg-headers-handler"},{"name":"prefill-filter","parameters":null,"type":"prefill-filter"},{"name":"decode-filter","parameters":null,"type":"decode-filter"},{"name":"queue-scorer","parameters":null,"type":"queue-scorer"},{"name":"kv-cache-utilization-scorer","parameters":null,"type":"kv-cache-utilization-scorer"},{"name":"active-request-scorer","parameters":null,"type":"active-request-scorer"},{"name":"prefix-cache-scorer","parameters":null,"type":"prefix-cache-scorer"},{"name":"max-score-picker","parameters":null,"type":"max-score-picker"},{"name":"always-disagg-pd-decider","parameters":null,"type":"always-disagg-pd-decider"},{"name":"disagg-profile-handler","parameters":{"deciders":{"prefill":"always-disagg-pd-decider"}},"type":"disagg-profile-handler"},{"name":"metrics-data-source","parameters":{"scheme":"https"},"type":"metrics-data-source"}],"schedulingProfiles":[{"name":"prefill","plugins":[{"pluginRef":"prefill-filter","weight":null},{"pluginRef":"prefix-cache-scorer","weight":3},{"pluginRef":"queue-scorer","weight":2},{"pluginRef":"kv-cache-utilization-scorer","weight":2},{"pluginRef":"max-score-picker","weight":null}]},{"name":"decode","plugins":[{"pluginRef":"decode-filter","weight":null},{"pluginRef":"active-request-scorer","weight":2},{"pluginRef":"prefix-cache-scorer","weight":3},{"pluginRef":"max-score-picker","weight":null}]}]}} [e2e-llm-inference-service] {"level":"info","ts":1785262382.120239,"caller":"disagg/disagg_profile_handler.go:186","msg":"No deciders.encode configured, E disaggregation disabled"} [e2e-llm-inference-service] {"level":"info","ts":1785262382.120313,"logger":"utilization-detector/utilization-detector","caller":"utilization/detector.go:83","msg":"Creating new UtilizationDetector","queueDepthThreshold":5,"kvCacheUtilThreshold":0.8,"metricsStalenessThreshold":"200ms","headroom":0} [e2e-llm-inference-service] {"level":"info","ts":1785262382.120369,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"vllm","mapping":"Mapping{all specs enabled}"} [e2e-llm-inference-service] {"level":"info","ts":1785262382.1204007,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"sglang","mapping":"Mapping{disabled: [lora]}"} [e2e-llm-inference-service] {"level":"info","ts":1785262382.1204364,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"trtllm-serve","mapping":"Mapping{disabled: [lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1785262382.120494,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"triton-tensorrt-llm","mapping":"Mapping{disabled: [lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1785262382.120513,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"triton","mapping":"Mapping{disabled: [kv, lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1785262382.1205835,"caller":"loader/configloader.go:154","msg":"Instantiated all plugins and applied system defaults. Effective raw configuration","config":"{Plugins: [{Name: disagg-headers-handler, Type: disagg-headers-handler} {Name: prefill-filter, Type: prefill-filter} {Name: decode-filter, Type: decode-filter} {Name: queue-scorer, Type: queue-scorer} {Name: kv-cache-utilization-scorer, Type: kv-cache-utilization-scorer} {Name: active-request-scorer, Type: active-request-scorer} {Name: prefix-cache-scorer, Type: prefix-cache-scorer} {Name: max-score-picker, Type: max-score-picker} {Name: always-disagg-pd-decider, Type: always-disagg-pd-decider} {Name: disagg-profile-handler, Type: disagg-profile-handler, Parameters: {\"deciders\":{\"prefill\":\"always-disagg-pd-decider\"}}} {Name: metrics-data-source, Type: metrics-data-source, Parameters: {\"scheme\":\"https\"}} {Name: fcfs-ordering-policy, Type: fcfs-ordering-policy} {Name: global-strict-fairness-policy, Type: global-strict-fairness-policy} {Name: static-usage-limit-policy, Type: static-usage-limit-policy} {Name: openai-parser, Type: openai-parser} {Name: anthropic-parser, Type: anthropic-parser} {Name: vllmhttp-parser, Type: vllmhttp-parser} {Name: utilization-detector, Type: utilization-detector} {Name: core-metrics-extractor, Type: core-metrics-extractor}], SchedulingProfiles: [{Name: prefill, Plugins: [{PluginRef: prefill-filter} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: queue-scorer, Weight: 2.00} {PluginRef: kv-cache-utilization-scorer, Weight: 2.00} {PluginRef: max-score-picker}]} {Name: decode, Plugins: [{PluginRef: decode-filter} {PluginRef: active-request-scorer, Weight: 2.00} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: max-score-picker}]}], DataLayer: {Sources: [{PluginRef: metrics-data-source, Extractors: [{PluginRef: core-metrics-extractor}]}], Discovery: }, FlowControl: {MaxBytes: unlimited, MaxRequests: unlimited, SaturationDetector: {PluginRef: utilization-detector}}, RequestHandler: {Parsers: [{PluginRef: openai-parser}, {PluginRef: anthropic-parser}, {PluginRef: vllmhttp-parser}]}}"} [e2e-llm-inference-service] {"level":"info","ts":1785262382.1206572,"caller":"datalayer/data_graph.go:116","msg":"auto-created default producer","producer":"inflight-load-producer/inflight-load-producer","dataKey":"InFlightLoadDataKey/inflight-load-producer","consumer":"active-request-scorer"} [e2e-llm-inference-service] {"level":"info","ts":1785262382.1206777,"caller":"approximateprefix/plugin.go:88","msg":"Prefix DataProducer initialized","config":{"autoTune":true,"blockSizeTokens":16,"blockSize":0,"maxPrefixBlocksToMatch":2048,"maxPrefixTokensToMatch":131072,"lruCapacityPerServer":31250}} [e2e-llm-inference-service] {"level":"info","ts":1785262382.120732,"caller":"approximateprefix/plugin.go:111","msg":"WARNING: configured blockSizeTokens is below the recommended minimum, overriding it.","blockSizeTokens":16,"minimum":64,"issue":"https://github.com/llm-d/llm-d-router/issues/1158"} [e2e-llm-inference-service] {"level":"info","ts":1785262382.120761,"caller":"datalayer/data_graph.go:116","msg":"auto-created default producer","producer":"approx-prefix-cache-producer/approx-prefix-cache-producer","dataKey":"PrefixCacheMatchInfoDataKey/approx-prefix-cache-producer","consumer":"prefix-cache-scorer"} [e2e-llm-inference-service] {"level":"info","ts":1785262382.1207747,"caller":"datalayer/data_graph.go:116","msg":"auto-created default producer","producer":"token-producer/token-producer","dataKey":"TokenizedPrompt/token-producer","consumer":"disagg-profile-handler"} [e2e-llm-inference-service] {"level":"info","ts":1785262382.120887,"caller":"runner/runner.go:685","msg":"loaded configuration from file/text successfully"} [e2e-llm-inference-service] {"level":"info","ts":1785262382.1208966,"logger":"setup","caller":"runner/runner.go:308","msg":"EPP config after phase two","config":"{SchedulerConfig:{ProfileHandler: disagg-profile-handler/disagg-profile-handler, Profiles: map[decode:{Filters: [decode-filter/by-label], Scorers: [active-request-scorer/active-request-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000], Picker: max-score-picker/max-score-picker} prefill:{Filters: [prefill-filter/by-label], Scorers: [prefix-cache-scorer/prefix-cache-scorer: 3.000000, queue-scorer/queue-scorer: 2.000000, kv-cache-utilization-scorer/kv-cache-utilization-scorer: 2.000000], Picker: max-score-picker/max-score-picker}]} SaturationDetector:0xc0007c3300 DataConfig:{Sources:[{Plugin:0xc0008ba6c0 Extractors:[0xc0007c3500]}]} FlowControlConfig: ParserRegistry:0xc0007c3bc0}"} [e2e-llm-inference-service] {"level":"info","ts":1785262382.137345,"logger":"setup","caller":"runner/runner.go:352","msg":"Setting pprof handlers"} [e2e-llm-inference-service] {"level":"info","ts":1785262382.13737,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/profile"} [e2e-llm-inference-service] {"level":"info","ts":1785262382.1373844,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/trace"} [e2e-llm-inference-service] {"level":"info","ts":1785262382.1373887,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/goroutine"} [e2e-llm-inference-service] {"level":"info","ts":1785262382.1373935,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/allocs"} [e2e-llm-inference-service] {"level":"info","ts":1785262382.137398,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/"} [e2e-llm-inference-service] {"level":"info","ts":1785262382.137402,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/symbol"} [e2e-llm-inference-service] {"level":"info","ts":1785262382.1374059,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/heap"} [e2e-llm-inference-service] {"level":"info","ts":1785262382.1374104,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/threadcreate"} [e2e-llm-inference-service] {"level":"info","ts":1785262382.1374147,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/block"} [e2e-llm-inference-service] {"level":"info","ts":1785262382.1374187,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/mutex"} [e2e-llm-inference-service] {"level":"info","ts":1785262382.1374228,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/cmdline"} [e2e-llm-inference-service] {"level":"info","ts":1785262382.137434,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/plugins/state"} [e2e-llm-inference-service] {"level":"info","ts":1785262382.137439,"logger":"setup","caller":"runner/runner.go:373","msg":"parsed config","scheduler-config":"{ProfileHandler: disagg-profile-handler/disagg-profile-handler, Profiles: map[decode:{Filters: [decode-filter/by-label], Scorers: [active-request-scorer/active-request-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000], Picker: max-score-picker/max-score-picker} prefill:{Filters: [prefill-filter/by-label], Scorers: [prefix-cache-scorer/prefix-cache-scorer: 3.000000, queue-scorer/queue-scorer: 2.000000, kv-cache-utilization-scorer/kv-cache-utilization-scorer: 2.000000], Picker: max-score-picker/max-score-picker}]}"} [e2e-llm-inference-service] {"level":"info","ts":1785262382.1374698,"logger":"setup","caller":"datalayer/runtime.go:99","msg":"Configuring datalayer runtime","numSources":1} [e2e-llm-inference-service] {"level":"info","ts":1785262382.1374774,"logger":"setup","caller":"datalayer/runtime.go:118","msg":"Processing source","source":"metrics-data-source","numExtractors":1} [e2e-llm-inference-service] {"level":"info","ts":1785262382.1374905,"logger":"setup","caller":"datalayer/runtime.go:147","msg":"Source configured","source":"metrics-data-source","extractors":["core-metrics-extractor/core-metrics-extractor"]} [e2e-llm-inference-service] {"level":"info","ts":1785262382.1375098,"logger":"setup","caller":"datalayer/runtime.go:206","msg":"Datalayer runtime configured","pollers":1,"notifiers":0,"endpointSources":1} [e2e-llm-inference-service] {"level":"info","ts":1785262382.1375203,"logger":"setup","caller":"runner/runner.go:833","msg":"Experimental Flow Control layer is disabled, using legacy admission control"} [e2e-llm-inference-service] {"level":"info","ts":1785262382.1376243,"logger":"setup","caller":"runner/runner.go:721","msg":"ExtProc server runner added to manager."} [e2e-llm-inference-service] {"level":"info","ts":1785262382.1376367,"logger":"setup","caller":"runner/runner.go:260","msg":"Controller manager starting"} [e2e-llm-inference-service] {"level":"info","ts":1785262382.1376698,"logger":"controller-runtime.metrics","caller":"server/server.go:208","msg":"Starting metrics server"} [e2e-llm-inference-service] {"level":"info","ts":1785262382.1378486,"caller":"runnable/grpc.go:35","msg":"gRPC server starting","name":"health"} [e2e-llm-inference-service] {"level":"info","ts":1785262382.138035,"caller":"runnable/grpc.go:43","msg":"gRPC server listening","name":"health","port":9003} [e2e-llm-inference-service] {"level":"info","ts":1785262382.138164,"logger":"controller-runtime.metrics","caller":"server/server.go:247","msg":"Serving metrics server","bindAddress":":9090","secure":false} [e2e-llm-inference-service] {"level":"info","ts":1785262382.1381686,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","source":"kind source: *v1.InferencePool"} [e2e-llm-inference-service] {"level":"info","ts":1785262382.1384358,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferenceobjective","controllerGroup":"llm-d.ai","controllerKind":"InferenceObjective","source":"kind source: *v1alpha2.InferenceObjective"} [e2e-llm-inference-service] {"level":"info","ts":1785262382.1385403,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"pod","controllerGroup":"","controllerKind":"Pod","source":"kind source: *v1.Pod"} [e2e-llm-inference-service] {"level":"info","ts":1785262382.1387985,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferencemodelrewrite","controllerGroup":"llm-d.ai","controllerKind":"InferenceModelRewrite","source":"kind source: *v1alpha2.InferenceModelRewrite"} [e2e-llm-inference-service] {"level":"info","ts":1785262382.1389713,"caller":"runnable/grpc.go:35","msg":"gRPC server starting","name":"ext-proc"} [e2e-llm-inference-service] {"level":"info","ts":1785262382.1390865,"caller":"runnable/grpc.go:43","msg":"gRPC server listening","name":"ext-proc","port":9002} [e2e-llm-inference-service] {"level":"info","ts":1785262382.1418686,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.InferenceObjective","reflector":"go/pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1785262382.1421678,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.InferenceModelRewrite","reflector":"go/pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1785262382.1423442,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.InferencePool","reflector":"go/pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1785262382.1428702,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Pod","reflector":"go/pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1785262382.2399504,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferencemodelrewrite","controllerGroup":"llm-d.ai","controllerKind":"InferenceModelRewrite"} [e2e-llm-inference-service] {"level":"info","ts":1785262382.239986,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferencemodelrewrite","controllerGroup":"llm-d.ai","controllerKind":"InferenceModelRewrite","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1785262382.2399487,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferenceobjective","controllerGroup":"llm-d.ai","controllerKind":"InferenceObjective"} [e2e-llm-inference-service] {"level":"info","ts":1785262382.2400134,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferenceobjective","controllerGroup":"llm-d.ai","controllerKind":"InferenceObjective","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1785262382.3390422,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool"} [e2e-llm-inference-service] {"level":"info","ts":1785262382.339073,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1785262382.3392289,"caller":"controller/inferencepool_reconciler.go:46","msg":"Reconciling InferencePool","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool","namespace":"e2e-test-llm-inference-service-48639af5"},"namespace":"e2e-test-llm-inference-service-48639af5","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool","reconcileID":"37a18333-e16c-4ee3-a33c-a08fb4951a60"} [e2e-llm-inference-service] {"level":"info","ts":1785262382.3401902,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"pod","controllerGroup":"","controllerKind":"Pod"} [e2e-llm-inference-service] {"level":"info","ts":1785262382.3402112,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"pod","controllerGroup":"","controllerKind":"Pod","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1785262390.5715032,"caller":"controller/inferencepool_reconciler.go:46","msg":"Reconciling InferencePool","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool","namespace":"e2e-test-llm-inference-service-48639af5"},"namespace":"e2e-test-llm-inference-service-48639af5","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool","reconcileID":"b715d943-16b1-43cc-b324-e96c2c0660c0"} [e2e-llm-inference-service] {"level":"info","ts":1785262546.2642965,"caller":"inflightload/producer.go:308","msg":"Injected dynamic attribute into endpoint","key":"InFlightLoadDataKey/inflight-load-producer","endpoint":"e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd-rank-0"} [e2e-llm-inference-service] {"level":"info","ts":1785262546.264374,"caller":"controller/pod_reconciler.go:99","msg":"Pod already exists","controller":"pod","controllerGroup":"","controllerKind":"Pod","Pod":{"name":"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd","namespace":"e2e-test-llm-inference-service-48639af5"},"namespace":"e2e-test-llm-inference-service-48639af5","name":"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd","reconcileID":"28a67e8f-e92a-4732-a8d5-d3fd916b5711"} [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-service [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 9d75ad33-be7a-4945-99c8-20aefcd8f922 [e2e-llm-inference-service] resourceVersion: '64038' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-28T18:13:33Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T18:13:34Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - ip: 10.134.0.49 [e2e-llm-inference-service] nodeName: ip-10-0-135-20.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche6mt66 [e2e-llm-inference-service] uid: 150a3f51-fd04-40ce-84ce-801888b42c71 [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 8c985dc9-332f-4592-b134-69fab8e9030a [e2e-llm-inference-service] resourceVersion: '65424' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-28T18:15:46Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T18:15:46Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - ip: 10.132.0.61 [e2e-llm-inference-service] nodeName: ip-10-0-135-188.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd [e2e-llm-inference-service] uid: 714f9546-db67-453c-b8d0-f91fdaa6a01e [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c4mdcs [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 50febc0e-e107-4f9e-9767-670777188d7c [e2e-llm-inference-service] resourceVersion: '63499' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 6d9fffb56c [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.134.0.48/23"],"mac_address":"0a:58:0a:86:00:30","gateway_ips":["10.134.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.134.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.134.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.134.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.134.0.1"}],"ip_address":"10.134.0.48/23","gateway_ip":"10.134.0.1","role":"primary"}}' [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c [e2e-llm-inference-service] uid: 9252e7ae-b0b0-469f-a93f-e864ee3131dc [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-135-20 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"9252e7ae-b0b0-469f-a93f-e864ee3131dc"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8001,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"llm-d-routing-sidecar"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAMESPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:initContainerStatuses: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] - name: kube-api-access-mrwb6 [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/pd-sidecar [e2e-llm-inference-service] - --port=8000 [e2e-llm-inference-service] - --vllm-port=8001 [e2e-llm-inference-service] - --kv-connector=nixlv2 [e2e-llm-inference-service] - --enable-ssrf-protection=true [e2e-llm-inference-service] - --pool-group=inference.networking.x-k8s.io [e2e-llm-inference-service] - --inference-pool=e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] - --secure-proxy=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - --enable-tls=decoder [e2e-llm-inference-service] - --enable-tls=prefiller [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: INFERENCE_POOL_NAMESPACE [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] - name: INFERENCE_POOL_NAME [e2e-llm-inference-service] value: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kube-api-access-mrwb6 [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000970000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to infer\ [e2e-llm-inference-service] \ RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null\n\ [e2e-llm-inference-service] \ grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/* 2>/dev/null\n\ [e2e-llm-inference-service] \n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"$hca_dir\"\ [e2e-llm-inference-service] \ ]; then\n hca_name=$(basename \"$hca_dir\")\n port_state_file=\"\ [e2e-llm-inference-service] $hca_dir/ports/1/state\" # Assume port 1\n type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\ [e2e-llm-inference-service] \n\n echo \"[Infer RoCE] Check if the port state file ${port_state_file}\ [e2e-llm-inference-service] \ exists and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] &&\ [e2e-llm-inference-service] \ grep -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found active\ [e2e-llm-inference-service] \ HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n else\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Skipping inactive or down HCA: $hca_name\"\ [e2e-llm-inference-service] \n fi\n fi\n done\n\n # Check if we found any active HCAs\n\ [e2e-llm-inference-service] \ if [ ${#active_hcas[@]} -gt 0 ]; then\n # Join the array elements with\ [e2e-llm-inference-service] \ a comma\n hca_port_pairs=()\n for hca in \"${active_hcas[@]}\";\ [e2e-llm-inference-service] \ do\n hca_port_pairs+=(\"${hca}:1\")\n done\n\n active_hca_list=$(IFS=,;\ [e2e-llm-inference-service] \ echo \"${active_hcas[*]}\")\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\"\ [e2e-llm-inference-service] )\n echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n\ [e2e-llm-inference-service] \ export UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found. NCCL_IB_HCA\ [e2e-llm-inference-service] \ will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Finding GID_INDEX for each active HCA (SR-IOV compatible)...\"\ [e2e-llm-inference-service] \n\n # For SR-IOV environments, find the most common IPv4 RoCE v2 GID index\ [e2e-llm-inference-service] \ across all HCAs\n declare -A gid_index_count\n declare -A hca_gid_index\n\ [e2e-llm-inference-service] \n for hca_name in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Processing HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for\ [e2e-llm-inference-service] \ this HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"$tpath\"\ [e2e-llm-inference-service] \ 2>/dev/null; then\n idx=$(basename \"$tpath\")\n \ [e2e-llm-inference-service] \ gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n \ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo \"\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Found IPv4 RoCE v2 GID for ${hca_name}:\ [e2e-llm-inference-service] \ index=${idx}, gid=${gid_value}\"\n hca_gid_index[\"${hca_name}\"\ [e2e-llm-inference-service] ]=\"${idx}\"\n gid_index_count[\"${idx}\"]=$((${gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]} + 1))\n break # Use first found IPv4 GID per\ [e2e-llm-inference-service] \ HCA\n fi\n fi\n done\n done\n\n\ [e2e-llm-inference-service] \ # Find the most common GID index (most likely to be consistent across\ [e2e-llm-inference-service] \ nodes)\n best_gid_index=\"\"\n max_count=0\n for idx in \"\ [e2e-llm-inference-service] ${!gid_index_count[@]}\"; do\n count=${gid_index_count[\"${idx}\"]}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n \ [e2e-llm-inference-service] \ if [ $count -gt $max_count ]; then\n max_count=$count\n\ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n #\ [e2e-llm-inference-service] \ Use deterministic fallback if tied - prefer index 3 (SR-IOV standard)\n \ [e2e-llm-inference-service] \ if [ ${#gid_index_count[@]} -gt 1 ]; then\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Multiple GID indices found, selecting most common: ${best_gid_index}\"\n \ [e2e-llm-inference-service] \ # If there's a tie, prefer index 3 as it's most common in SR-IOV setups\n\ [e2e-llm-inference-service] \ if [ -n \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\"\ [e2e-llm-inference-service] \ -eq \"$max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for NCCL,\ [e2e-llm-inference-service] \ NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR: No valid\ [e2e-llm-inference-service] \ IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any HCA.\"\n \ [e2e-llm-inference-service] \ fi\n else\n echo \"[Infer RoCE] No active HCAs found, skipping GID_INDEX\ [e2e-llm-inference-service] \ inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints landed in vLLM\ [e2e-llm-inference-service] \ 0.16.0 (vllm-project/vllm#30011).\n# Older versions still need the blanket\ [e2e-llm-inference-service] \ --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+ ]] &&\ [e2e-llm-inference-service] \ [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort -V | head\ [e2e-llm-inference-service] \ -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout 40\"\ [e2e-llm-inference-service] \nfi\n\n# --kv-transfer-config with OffloadingConnector requires vLLM 0.22.0+\ [e2e-llm-inference-service] \ (vllm-project/vllm#40020).\nKV_TRANSFER_ARGS=\"\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.22.0\" ]; then\n if [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\" != *\"\ [e2e-llm-inference-service] --kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"* ]] &&\ [e2e-llm-inference-service] \ [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n KV_TRANSFER_ARGS=\"\"\ [e2e-llm-inference-service] \n fi\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name \"\ [e2e-llm-inference-service] facebook/opt-125m\" \"publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8001 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n ${KV_TRANSFER_ARGS} \\\n \\\n --enable-ssl-refresh \\\n --ssl-certfile\ [e2e-llm-inference-service] \ /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8001 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-mrwb6 [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000970000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] nodeName: ip-10-0-135-20.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c31,c20 [e2e-llm-inference-service] fsGroup: 1000970000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-w6gcf [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-2wqp2 [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Pending [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] reason: ContainersNotInitialized [e2e-llm-inference-service] message: 'containers with incomplete status: [llm-d-routing-sidecar]' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] reason: ContainersNotReady [e2e-llm-inference-service] message: 'containers with unready status: [llm-d-routing-sidecar main]' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] reason: ContainersNotReady [e2e-llm-inference-service] message: 'containers with unready status: [llm-d-routing-sidecar main]' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] hostIP: 10.0.135.20 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.135.20 [e2e-llm-inference-service] startTime: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] initContainerStatuses: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] state: [e2e-llm-inference-service] waiting: [e2e-llm-inference-service] reason: PodInitializing [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: false [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.9.0 [e2e-llm-inference-service] imageID: '' [e2e-llm-inference-service] started: false [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-mrwb6 [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] waiting: [e2e-llm-inference-service] reason: PodInitializing [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: false [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] imageID: '' [e2e-llm-inference-service] started: false [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-mrwb6 [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b89cd7957- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 714f9546-db67-453c-b8d0-f91fdaa6a01e [e2e-llm-inference-service] resourceVersion: '65420' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 7b89cd7957 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.132.0.61/23"],"mac_address":"0a:58:0a:84:00:3d","gateway_ips":["10.132.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.132.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.132.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.132.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.132.0.1"}],"ip_address":"10.132.0.61/23","gateway_ip":"10.132.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.132.0.61\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:84:00:3d\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b89cd7957 [e2e-llm-inference-service] uid: cfdbcd01-fb05-4f31-9c0c-15f65638c252 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-135-188 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"cfdbcd01-fb05-4f31-9c0c-15f65638c252"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T18:13:06Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T18:15:46Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.132.0.61"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] - name: kube-api-access-h6ss2 [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to infer\ [e2e-llm-inference-service] \ RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null\n\ [e2e-llm-inference-service] \ grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/* 2>/dev/null\n\ [e2e-llm-inference-service] \n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"$hca_dir\"\ [e2e-llm-inference-service] \ ]; then\n hca_name=$(basename \"$hca_dir\")\n port_state_file=\"\ [e2e-llm-inference-service] $hca_dir/ports/1/state\" # Assume port 1\n type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\ [e2e-llm-inference-service] \n\n echo \"[Infer RoCE] Check if the port state file ${port_state_file}\ [e2e-llm-inference-service] \ exists and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] &&\ [e2e-llm-inference-service] \ grep -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found active\ [e2e-llm-inference-service] \ HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n else\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Skipping inactive or down HCA: $hca_name\"\ [e2e-llm-inference-service] \n fi\n fi\n done\n\n # Check if we found any active HCAs\n\ [e2e-llm-inference-service] \ if [ ${#active_hcas[@]} -gt 0 ]; then\n # Join the array elements with\ [e2e-llm-inference-service] \ a comma\n hca_port_pairs=()\n for hca in \"${active_hcas[@]}\";\ [e2e-llm-inference-service] \ do\n hca_port_pairs+=(\"${hca}:1\")\n done\n\n active_hca_list=$(IFS=,;\ [e2e-llm-inference-service] \ echo \"${active_hcas[*]}\")\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\"\ [e2e-llm-inference-service] )\n echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n\ [e2e-llm-inference-service] \ export UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found. NCCL_IB_HCA\ [e2e-llm-inference-service] \ will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Finding GID_INDEX for each active HCA (SR-IOV compatible)...\"\ [e2e-llm-inference-service] \n\n # For SR-IOV environments, find the most common IPv4 RoCE v2 GID index\ [e2e-llm-inference-service] \ across all HCAs\n declare -A gid_index_count\n declare -A hca_gid_index\n\ [e2e-llm-inference-service] \n for hca_name in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Processing HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for\ [e2e-llm-inference-service] \ this HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"$tpath\"\ [e2e-llm-inference-service] \ 2>/dev/null; then\n idx=$(basename \"$tpath\")\n \ [e2e-llm-inference-service] \ gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n \ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo \"\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Found IPv4 RoCE v2 GID for ${hca_name}:\ [e2e-llm-inference-service] \ index=${idx}, gid=${gid_value}\"\n hca_gid_index[\"${hca_name}\"\ [e2e-llm-inference-service] ]=\"${idx}\"\n gid_index_count[\"${idx}\"]=$((${gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]} + 1))\n break # Use first found IPv4 GID per\ [e2e-llm-inference-service] \ HCA\n fi\n fi\n done\n done\n\n\ [e2e-llm-inference-service] \ # Find the most common GID index (most likely to be consistent across\ [e2e-llm-inference-service] \ nodes)\n best_gid_index=\"\"\n max_count=0\n for idx in \"\ [e2e-llm-inference-service] ${!gid_index_count[@]}\"; do\n count=${gid_index_count[\"${idx}\"]}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n \ [e2e-llm-inference-service] \ if [ $count -gt $max_count ]; then\n max_count=$count\n\ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n #\ [e2e-llm-inference-service] \ Use deterministic fallback if tied - prefer index 3 (SR-IOV standard)\n \ [e2e-llm-inference-service] \ if [ ${#gid_index_count[@]} -gt 1 ]; then\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Multiple GID indices found, selecting most common: ${best_gid_index}\"\n \ [e2e-llm-inference-service] \ # If there's a tie, prefer index 3 as it's most common in SR-IOV setups\n\ [e2e-llm-inference-service] \ if [ -n \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\"\ [e2e-llm-inference-service] \ -eq \"$max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for NCCL,\ [e2e-llm-inference-service] \ NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR: No valid\ [e2e-llm-inference-service] \ IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any HCA.\"\n \ [e2e-llm-inference-service] \ fi\n else\n echo \"[Infer RoCE] No active HCAs found, skipping GID_INDEX\ [e2e-llm-inference-service] \ inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints landed in vLLM\ [e2e-llm-inference-service] \ 0.16.0 (vllm-project/vllm#30011).\n# Older versions still need the blanket\ [e2e-llm-inference-service] \ --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+ ]] &&\ [e2e-llm-inference-service] \ [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort -V | head\ [e2e-llm-inference-service] \ -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout 40\"\ [e2e-llm-inference-service] \nfi\n\n# --kv-transfer-config with OffloadingConnector requires vLLM 0.22.0+\ [e2e-llm-inference-service] \ (vllm-project/vllm#40020).\nKV_TRANSFER_ARGS=\"\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.22.0\" ]; then\n if [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\" != *\"\ [e2e-llm-inference-service] --kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"* ]] &&\ [e2e-llm-inference-service] \ [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n KV_TRANSFER_ARGS=\"\"\ [e2e-llm-inference-service] \n fi\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name \"\ [e2e-llm-inference-service] facebook/opt-125m\" \\\n --port 8000 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n ${KV_TRANSFER_ARGS} \\\n \\\n --enable-ssl-refresh \\\n --ssl-certfile\ [e2e-llm-inference-service] \ /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-h6ss2 [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: File [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000970000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: default [e2e-llm-inference-service] serviceAccount: default [e2e-llm-inference-service] nodeName: ip-10-0-135-188.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c31,c20 [e2e-llm-inference-service] fsGroup: 1000970000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-w6gcf [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T18:13:37Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T18:15:46Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T18:15:46Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] hostIP: 10.0.135.188 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.135.188 [e2e-llm-inference-service] podIP: 10.132.0.61 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.132.0.61 [e2e-llm-inference-service] startTime: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-07-28T18:13:37Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: docker.io/vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] imageID: docker.io/vllm/vllm-openai-cpu@sha256:a8257c201fd2f696c146615987be61579aa2052367b73c27bd3fb1ab8e90bd4b [e2e-llm-inference-service] containerID: cri-o://b33819149f394b955aab6820246e62d181de5c977cd84d096a9ecb6caef8a88a [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-h6ss2 [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000970000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000970000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche6mt66 [e2e-llm-inference-service] generateName: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-7fc69df7b8- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 150a3f51-fd04-40ce-84ce-801888b42c71 [e2e-llm-inference-service] resourceVersion: '64036' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 7fc69df7b8 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.10.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.134.0.49/23"],"mac_address":"0a:58:0a:86:00:31","gateway_ips":["10.134.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.134.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.134.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.134.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.134.0.1"}],"ip_address":"10.134.0.49/23","gateway_ip":"10.134.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.134.0.49\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:86:00:31\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-7fc69df7b8 [e2e-llm-inference-service] uid: 9de6629b-16a6-4741-a10c-ad7f130c7643 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-135-20 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"9de6629b-16a6-4741-a10c-ad7f130c7643"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T18:13:02Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T18:13:34Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.134.0.49"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kube-api-access-s7bqn [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: llm-d.ai/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n- type:\ [e2e-llm-inference-service] \ disagg-headers-handler\n- type: prefill-filter\n- type: decode-filter\n- type:\ [e2e-llm-inference-service] \ queue-scorer\n- type: kv-cache-utilization-scorer\n- type: active-request-scorer\n\ [e2e-llm-inference-service] - type: prefix-cache-scorer\n- type: max-score-picker\n- type: always-disagg-pd-decider\n\ [e2e-llm-inference-service] - parameters:\n deciders:\n prefill: always-disagg-pd-decider\n type:\ [e2e-llm-inference-service] \ disagg-profile-handler\n- parameters:\n scheme: https\n type: metrics-data-source\n\ [e2e-llm-inference-service] schedulingProfiles:\n- name: prefill\n plugins:\n - pluginRef: prefill-filter\n\ [e2e-llm-inference-service] \ - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: queue-scorer\n\ [e2e-llm-inference-service] \ weight: 2\n - pluginRef: kv-cache-utilization-scorer\n weight: 2\n\ [e2e-llm-inference-service] \ - pluginRef: max-score-picker\n- name: decode\n plugins:\n - pluginRef:\ [e2e-llm-inference-service] \ decode-filter\n - pluginRef: active-request-scorer\n weight: 2\n - pluginRef:\ [e2e-llm-inference-service] \ prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kube-api-access-s7bqn [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000970000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] nodeName: ip-10-0-135-20.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c31,c20 [e2e-llm-inference-service] fsGroup: 1000970000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-w6gcf [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-82p9w [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T18:13:02Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T18:13:33Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T18:13:33Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] hostIP: 10.0.135.20 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.135.20 [e2e-llm-inference-service] podIP: 10.134.0.49 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.134.0.49 [e2e-llm-inference-service] startTime: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-07-28T18:13:02Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-llm-d-router-endpoint-picker@sha256:7fd1d61cb5505e026eebab0b9653758779c3dee241e37bea92a0bab961a48c50 [e2e-llm-inference-service] containerID: cri-o://6cf5e48b4e4fbf44d539e6220f03fc5333bb7d66fa5bd7584b6ef4d9d796c712 [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-s7bqn [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000970000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000970000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: c66bcd5f-16d8-41d9-be4b-9c4bb91e5f68 [e2e-llm-inference-service] resourceVersion: '63479' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] openshift.io/internal-registry-pull-secret-ref: llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-82p9w [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: c0bf7073-efea-42d3-83eb-e33d2c9137c1 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: openshift.io/image-registry-pull-secrets_service-account-controller [e2e-llm-inference-service] operation: Apply [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:imagePullSecrets: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:openshift.io/internal-registry-pull-secret-ref: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] k:{"name":"llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-82p9w"}: {} [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"c0bf7073-efea-42d3-83eb-e33d2c9137c1"}: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"default-dockercfg-w6gcf"}: {} [e2e-llm-inference-service] k:{"name":"seaweedfs-s3-creds"}: {} [e2e-llm-inference-service] secrets: [e2e-llm-inference-service] - name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: default-dockercfg-w6gcf [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-82p9w [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-w6gcf [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-82p9w [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: ServiceAccount [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 4fda3207-7d3e-48d1-90f9-dec9266573c8 [e2e-llm-inference-service] resourceVersion: '63447' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] openshift.io/internal-registry-pull-secret-ref: llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-2wqp2 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: c0bf7073-efea-42d3-83eb-e33d2c9137c1 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: openshift.io/image-registry-pull-secrets_service-account-controller [e2e-llm-inference-service] operation: Apply [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:imagePullSecrets: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:openshift.io/internal-registry-pull-secret-ref: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] k:{"name":"llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-2wqp2"}: {} [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"c0bf7073-efea-42d3-83eb-e33d2c9137c1"}: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"default-dockercfg-w6gcf"}: {} [e2e-llm-inference-service] k:{"name":"seaweedfs-s3-creds"}: {} [e2e-llm-inference-service] secrets: [e2e-llm-inference-service] - name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: default-dockercfg-w6gcf [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-2wqp2 [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-w6gcf [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-2wqp2 [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: ServiceAccount [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-service [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 4b9f076b-5402-4f5f-adbc-e23aa90ebec3 [e2e-llm-inference-service] resourceVersion: '63514' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: c0bf7073-efea-42d3-83eb-e33d2c9137c1 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"c0bf7073-efea-42d3-83eb-e33d2c9137c1"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] targetPort: grpc [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] targetPort: grpc-health [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] targetPort: metrics [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] targetPort: zmq [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] clusterIP: 172.31.170.35 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.170.35 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: c102e0ec-f910-4d9c-b0d0-33d1f82e1772 [e2e-llm-inference-service] resourceVersion: '63468' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: c0bf7073-efea-42d3-83eb-e33d2c9137c1 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"c0bf7073-efea-42d3-83eb-e33d2c9137c1"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] targetPort: 8000 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] clusterIP: 172.31.206.92 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.206.92 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 85ddb907-fa5c-4fb3-98c4-4c64ab1de8b5 [e2e-llm-inference-service] resourceVersion: '69855' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: c0bf7073-efea-42d3-83eb-e33d2c9137c1 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"c0bf7073-efea-42d3-83eb-e33d2c9137c1"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:rollingUpdate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:maxSurge: {} [e2e-llm-inference-service] f:maxUnavailable: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8001,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"llm-d-routing-sidecar"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAMESPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T18:23:02Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:unavailableReplicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/pd-sidecar [e2e-llm-inference-service] - --port=8000 [e2e-llm-inference-service] - --vllm-port=8001 [e2e-llm-inference-service] - --kv-connector=nixlv2 [e2e-llm-inference-service] - --enable-ssrf-protection=true [e2e-llm-inference-service] - --pool-group=inference.networking.x-k8s.io [e2e-llm-inference-service] - --inference-pool=e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] - --secure-proxy=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - --enable-tls=decoder [e2e-llm-inference-service] - --enable-tls=prefiller [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: INFERENCE_POOL_NAMESPACE [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] - name: INFERENCE_POOL_NAME [e2e-llm-inference-service] value: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n # Check if\ [e2e-llm-inference-service] \ we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ # Join the array elements with a comma\n hca_port_pairs=()\n \ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n hca_port_pairs+=(\"\ [e2e-llm-inference-service] ${hca}:1\")\n done\n\n active_hca_list=$(IFS=,; echo \"${active_hcas[*]}\"\ [e2e-llm-inference-service] )\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\n# --kv-transfer-config with OffloadingConnector requires vLLM\ [e2e-llm-inference-service] \ 0.22.0+ (vllm-project/vllm#40020).\nKV_TRANSFER_ARGS=\"\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.22.0\" ]; then\n if [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\" !=\ [e2e-llm-inference-service] \ *\"--kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"\ [e2e-llm-inference-service] * ]] && [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n KV_TRANSFER_ARGS=\"\ [e2e-llm-inference-service] \"\n fi\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \"publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8001 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n ${KV_TRANSFER_ARGS} \\\n \\\n --enable-ssl-refresh \\\n --ssl-certfile\ [e2e-llm-inference-service] \ /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8001 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: RollingUpdate [e2e-llm-inference-service] rollingUpdate: [e2e-llm-inference-service] maxUnavailable: 25% [e2e-llm-inference-service] maxSurge: 25% [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] unavailableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastUpdateTime: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastUpdateTime: '2026-07-28T18:23:02Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T18:23:02Z' [e2e-llm-inference-service] reason: ProgressDeadlineExceeded [e2e-llm-inference-service] message: ReplicaSet "llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c" [e2e-llm-inference-service] has timed out progressing. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: ea655d4a-70de-4f40-8031-f8d038839224 [e2e-llm-inference-service] resourceVersion: '65426' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: c0bf7073-efea-42d3-83eb-e33d2c9137c1 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"c0bf7073-efea-42d3-83eb-e33d2c9137c1"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:rollingUpdate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:maxSurge: {} [e2e-llm-inference-service] f:maxUnavailable: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T18:15:46Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n # Check if\ [e2e-llm-inference-service] \ we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ # Join the array elements with a comma\n hca_port_pairs=()\n \ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n hca_port_pairs+=(\"\ [e2e-llm-inference-service] ${hca}:1\")\n done\n\n active_hca_list=$(IFS=,; echo \"${active_hcas[*]}\"\ [e2e-llm-inference-service] )\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\n# --kv-transfer-config with OffloadingConnector requires vLLM\ [e2e-llm-inference-service] \ 0.22.0+ (vllm-project/vllm#40020).\nKV_TRANSFER_ARGS=\"\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.22.0\" ]; then\n if [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\" !=\ [e2e-llm-inference-service] \ *\"--kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"\ [e2e-llm-inference-service] * ]] && [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n KV_TRANSFER_ARGS=\"\ [e2e-llm-inference-service] \"\n fi\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \\\n --port 8000 \\\n ${ACCESS_LOG_ARGS} \\\n\ [e2e-llm-inference-service] \ ${SHUTDOWN_TIMEOUT_ARGS} \\\n ${KV_TRANSFER_ARGS} \\\n \\\n --enable-ssl-refresh\ [e2e-llm-inference-service] \ \\\n --ssl-certfile /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: File [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: RollingUpdate [e2e-llm-inference-service] rollingUpdate: [e2e-llm-inference-service] maxUnavailable: 25% [e2e-llm-inference-service] maxSurge: 25% [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-28T18:15:46Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T18:15:46Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-28T18:15:46Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b89cd7957" [e2e-llm-inference-service] has successfully progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 7b7ac3db-fe08-4ab7-864e-744ac2b02d8f [e2e-llm-inference-service] resourceVersion: '64041' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: c0bf7073-efea-42d3-83eb-e33d2c9137c1 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"c0bf7073-efea-42d3-83eb-e33d2c9137c1"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T18:13:34Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.10.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: llm-d.ai/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n-\ [e2e-llm-inference-service] \ type: disagg-headers-handler\n- type: prefill-filter\n- type: decode-filter\n\ [e2e-llm-inference-service] - type: queue-scorer\n- type: kv-cache-utilization-scorer\n- type: active-request-scorer\n\ [e2e-llm-inference-service] - type: prefix-cache-scorer\n- type: max-score-picker\n- type: always-disagg-pd-decider\n\ [e2e-llm-inference-service] - parameters:\n deciders:\n prefill: always-disagg-pd-decider\n\ [e2e-llm-inference-service] \ type: disagg-profile-handler\n- parameters:\n scheme: https\n type:\ [e2e-llm-inference-service] \ metrics-data-source\nschedulingProfiles:\n- name: prefill\n plugins:\n\ [e2e-llm-inference-service] \ - pluginRef: prefill-filter\n - pluginRef: prefix-cache-scorer\n \ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: queue-scorer\n weight: 2\n - pluginRef:\ [e2e-llm-inference-service] \ kv-cache-utilization-scorer\n weight: 2\n - pluginRef: max-score-picker\n\ [e2e-llm-inference-service] - name: decode\n plugins:\n - pluginRef: decode-filter\n - pluginRef:\ [e2e-llm-inference-service] \ active-request-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n\ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: Recreate [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-28T18:13:34Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T18:13:34Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-28T18:13:34Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-7fc69df7b8" [e2e-llm-inference-service] has successfully progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 9252e7ae-b0b0-469f-a93f-e864ee3131dc [e2e-llm-inference-service] resourceVersion: '63490' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 6d9fffb56c [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '2' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] uid: 85ddb907-fa5c-4fb3-98c4-4c64ab1de8b5 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"85ddb907-fa5c-4fb3-98c4-4c64ab1de8b5"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8001,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"llm-d-routing-sidecar"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAMESPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 6d9fffb56c [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 6d9fffb56c [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/pd-sidecar [e2e-llm-inference-service] - --port=8000 [e2e-llm-inference-service] - --vllm-port=8001 [e2e-llm-inference-service] - --kv-connector=nixlv2 [e2e-llm-inference-service] - --enable-ssrf-protection=true [e2e-llm-inference-service] - --pool-group=inference.networking.x-k8s.io [e2e-llm-inference-service] - --inference-pool=e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] - --secure-proxy=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - --enable-tls=decoder [e2e-llm-inference-service] - --enable-tls=prefiller [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: INFERENCE_POOL_NAMESPACE [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] - name: INFERENCE_POOL_NAME [e2e-llm-inference-service] value: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n # Check if\ [e2e-llm-inference-service] \ we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ # Join the array elements with a comma\n hca_port_pairs=()\n \ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n hca_port_pairs+=(\"\ [e2e-llm-inference-service] ${hca}:1\")\n done\n\n active_hca_list=$(IFS=,; echo \"${active_hcas[*]}\"\ [e2e-llm-inference-service] )\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\n# --kv-transfer-config with OffloadingConnector requires vLLM\ [e2e-llm-inference-service] \ 0.22.0+ (vllm-project/vllm#40020).\nKV_TRANSFER_ARGS=\"\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.22.0\" ]; then\n if [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\" !=\ [e2e-llm-inference-service] \ *\"--kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"\ [e2e-llm-inference-service] * ]] && [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n KV_TRANSFER_ARGS=\"\ [e2e-llm-inference-service] \"\n fi\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \"publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8001 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n ${KV_TRANSFER_ARGS} \\\n \\\n --enable-ssl-refresh \\\n --ssl-certfile\ [e2e-llm-inference-service] \ /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8001 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b89cd7957 [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: cfdbcd01-fb05-4f31-9c0c-15f65638c252 [e2e-llm-inference-service] resourceVersion: '65425' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 7b89cd7957 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '2' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill [e2e-llm-inference-service] uid: ea655d4a-70de-4f40-8031-f8d038839224 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"ea655d4a-70de-4f40-8031-f8d038839224"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T18:15:46Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 7b89cd7957 [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 7b89cd7957 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n # Check if\ [e2e-llm-inference-service] \ we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ # Join the array elements with a comma\n hca_port_pairs=()\n \ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n hca_port_pairs+=(\"\ [e2e-llm-inference-service] ${hca}:1\")\n done\n\n active_hca_list=$(IFS=,; echo \"${active_hcas[*]}\"\ [e2e-llm-inference-service] )\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\n# --kv-transfer-config with OffloadingConnector requires vLLM\ [e2e-llm-inference-service] \ 0.22.0+ (vllm-project/vllm#40020).\nKV_TRANSFER_ARGS=\"\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.22.0\" ]; then\n if [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\" !=\ [e2e-llm-inference-service] \ *\"--kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"\ [e2e-llm-inference-service] * ]] && [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n KV_TRANSFER_ARGS=\"\ [e2e-llm-inference-service] \"\n fi\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \\\n --port 8000 \\\n ${ACCESS_LOG_ARGS} \\\n\ [e2e-llm-inference-service] \ ${SHUTDOWN_TIMEOUT_ARGS} \\\n ${KV_TRANSFER_ARGS} \\\n \\\n --enable-ssl-refresh\ [e2e-llm-inference-service] \ \\\n --ssl-certfile /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: File [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-7fc69df7b8 [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 9de6629b-16a6-4741-a10c-ad7f130c7643 [e2e-llm-inference-service] resourceVersion: '64039' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 7fc69df7b8 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler [e2e-llm-inference-service] uid: 7b7ac3db-fe08-4ab7-864e-744ac2b02d8f [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7b7ac3db-fe08-4ab7-864e-744ac2b02d8f"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T18:13:34Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 7fc69df7b8 [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 7fc69df7b8 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.10.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: llm-d.ai/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n-\ [e2e-llm-inference-service] \ type: disagg-headers-handler\n- type: prefill-filter\n- type: decode-filter\n\ [e2e-llm-inference-service] - type: queue-scorer\n- type: kv-cache-utilization-scorer\n- type: active-request-scorer\n\ [e2e-llm-inference-service] - type: prefix-cache-scorer\n- type: max-score-picker\n- type: always-disagg-pd-decider\n\ [e2e-llm-inference-service] - parameters:\n deciders:\n prefill: always-disagg-pd-decider\n\ [e2e-llm-inference-service] \ type: disagg-profile-handler\n- parameters:\n scheme: https\n type:\ [e2e-llm-inference-service] \ metrics-data-source\nschedulingProfiles:\n- name: prefill\n plugins:\n\ [e2e-llm-inference-service] \ - pluginRef: prefill-filter\n - pluginRef: prefix-cache-scorer\n \ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: queue-scorer\n weight: 2\n - pluginRef:\ [e2e-llm-inference-service] \ kv-cache-utilization-scorer\n weight: 2\n - pluginRef: max-score-picker\n\ [e2e-llm-inference-service] - name: decode\n plugins:\n - pluginRef: decode-filter\n - pluginRef:\ [e2e-llm-inference-service] \ active-request-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n\ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-rb [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: b5ba3d2b-189b-470c-b7c2-707e063789a2 [e2e-llm-inference-service] resourceVersion: '63506' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: c0bf7073-efea-42d3-83eb-e33d2c9137c1 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"c0bf7073-efea-42d3-83eb-e33d2c9137c1"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] apiGroup: rbac.authorization.k8s.io [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-role [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-rb [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: cdf30aab-9a6b-4ed2-aa73-114fbcc91645 [e2e-llm-inference-service] resourceVersion: '63457' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: c0bf7073-efea-42d3-83eb-e33d2c9137c1 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"c0bf7073-efea-42d3-83eb-e33d2c9137c1"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] apiGroup: rbac.authorization.k8s.io [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-role [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-role [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 49bcf8d8-817a-4c88-bfe9-936a10f573f5 [e2e-llm-inference-service] resourceVersion: '63503' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: c0bf7073-efea-42d3-83eb-e33d2c9137c1 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"c0bf7073-efea-42d3-83eb-e33d2c9137c1"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - llm-d.ai [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] - create [e2e-llm-inference-service] - update [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - delete [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-role [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: a008a76c-4067-41a3-86a1-8709348f4f4d [e2e-llm-inference-service] resourceVersion: '63451' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: c0bf7073-efea-42d3-83eb-e33d2c9137c1 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"c0bf7073-efea-42d3-83eb-e33d2c9137c1"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-service-5m6ss [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-e8706282-epp-service- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: ca138f28-0607-46f1-9270-b00c1a64cd1e [e2e-llm-inference-service] resourceVersion: '64040' [e2e-llm-inference-service] generation: 3 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: llmisvc-model-pvc-router-manage-e8706282-epp-service [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-28T18:13:33Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-service [e2e-llm-inference-service] uid: 4b9f076b-5402-4f5f-adbc-e23aa90ebec3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T18:13:34Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"4b9f076b-5402-4f5f-adbc-e23aa90ebec3"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.134.0.49 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche6mt66 [e2e-llm-inference-service] uid: 150a3f51-fd04-40ce-84ce-801888b42c71 [e2e-llm-inference-service] nodeName: ip-10-0-135-20.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-strxnv [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 9efe0407-e02e-4d94-9d25-fd470539581c [e2e-llm-inference-service] resourceVersion: '65422' [e2e-llm-inference-service] generation: 3 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-28T18:15:46Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] uid: c102e0ec-f910-4d9c-b0d0-33d1f82e1772 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T18:15:46Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"c102e0ec-f910-4d9c-b0d0-33d1f82e1772"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.132.0.61 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd [e2e-llm-inference-service] uid: 714f9546-db67-453c-b8d0-f91fdaa6a01e [e2e-llm-inference-service] nodeName: ip-10-0-135-188.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-rb [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: b5ba3d2b-189b-470c-b7c2-707e063789a2 [e2e-llm-inference-service] resourceVersion: '63506' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: c0bf7073-efea-42d3-83eb-e33d2c9137c1 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"c0bf7073-efea-42d3-83eb-e33d2c9137c1"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] userNames: [e2e-llm-inference-service] - system:serviceaccount:e2e-test-llm-inference-service-48639af5:llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] groupNames: null [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-role [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-rb [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: cdf30aab-9a6b-4ed2-aa73-114fbcc91645 [e2e-llm-inference-service] resourceVersion: '63457' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: c0bf7073-efea-42d3-83eb-e33d2c9137c1 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"c0bf7073-efea-42d3-83eb-e33d2c9137c1"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] userNames: [e2e-llm-inference-service] - system:serviceaccount:e2e-test-llm-inference-service-48639af5:llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] groupNames: null [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-role [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-role [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 49bcf8d8-817a-4c88-bfe9-936a10f573f5 [e2e-llm-inference-service] resourceVersion: '63503' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: c0bf7073-efea-42d3-83eb-e33d2c9137c1 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"c0bf7073-efea-42d3-83eb-e33d2c9137c1"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - llm-d.ai [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - create [e2e-llm-inference-service] - delete [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - update [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-role [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: a008a76c-4067-41a3-86a1-8709348f4f4d [e2e-llm-inference-service] resourceVersion: '63451' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: c0bf7073-efea-42d3-83eb-e33d2c9137c1 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"c0bf7073-efea-42d3-83eb-e33d2c9137c1"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"c0bf7073-efea-42d3-83eb-e33d2c9137c1"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T18:13:10Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-28T18:13:10Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-28T18:13:12Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-route [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: c0bf7073-efea-42d3-83eb-e33d2c9137c1 [e2e-llm-inference-service] resourceVersion: '63785' [e2e-llm-inference-service] uid: 354a7a73-87f0-477d-9225-307425e96259 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T18:13:10Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T18:13:03Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"c0bf7073-efea-42d3-83eb-e33d2c9137c1"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T18:13:10Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-28T18:13:10Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-28T18:13:12Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-route [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: c0bf7073-efea-42d3-83eb-e33d2c9137c1 [e2e-llm-inference-service] resourceVersion: '63785' [e2e-llm-inference-service] uid: 354a7a73-87f0-477d-9225-307425e96259 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T18:13:10Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T18:13:03Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"c0bf7073-efea-42d3-83eb-e33d2c9137c1"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:endpointPickerRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:number: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:matchLabels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPorts: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-28T18:13:10Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: c0bf7073-efea-42d3-83eb-e33d2c9137c1 [e2e-llm-inference-service] resourceVersion: '63749' [e2e-llm-inference-service] uid: 450a9fdc-a61b-4234-b4d2-0c423b087a12 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] appProtocol: http [e2e-llm-inference-service] endpointPickerRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-service [e2e-llm-inference-service] port: [e2e-llm-inference-service] number: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPorts: [e2e-llm-inference-service] - number: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T18:13:10Z' [e2e-llm-inference-service] message: Referenced by an HTTPRoute accepted by the parentRef Gateway [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T18:13:10Z' [e2e-llm-inference-service] message: Referenced ExtensionRef resolved successfully [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: networking.istio.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] kind: AuthPolicy [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:13:03Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-policies [e2e-llm-inference-service] app.kubernetes.io/managed-by: odh-model-controller [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"c0bf7073-efea-42d3-83eb-e33d2c9137c1"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:rules: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:authentication: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:public: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:anonymous: {} [e2e-llm-inference-service] f:credentials: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:overrides: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fairness: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:response: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:success: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:headers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:x-gateway-inference-fairness-id: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:x-gateway-inference-objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:targetRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T18:13:03Z' [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Accepted"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Enforced"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-28T18:13:06Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-route-authn [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: c0bf7073-efea-42d3-83eb-e33d2c9137c1 [e2e-llm-inference-service] resourceVersion: '63694' [e2e-llm-inference-service] uid: 5837d2cf-41a9-45f8-9f0a-f30f07d9e907 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] rules: [e2e-llm-inference-service] authentication: [e2e-llm-inference-service] public: [e2e-llm-inference-service] anonymous: {} [e2e-llm-inference-service] credentials: {} [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] overrides: [e2e-llm-inference-service] fairness: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] objective: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] response: [e2e-llm-inference-service] success: [e2e-llm-inference-service] headers: [e2e-llm-inference-service] x-gateway-inference-fairness-id: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.fairness [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] x-gateway-inference-objective: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.objective [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-route [e2e-llm-inference-service] status: [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T18:13:04Z' [e2e-llm-inference-service] message: AuthPolicy has been accepted [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T18:13:06Z' [e2e-llm-inference-service] message: AuthPolicy has been successfully enforced [e2e-llm-inference-service] reason: Enforced [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Enforced [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:13:02Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"c0bf7073-efea-42d3-83eb-e33d2c9137c1"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T18:13:02Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-scheduler [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: c0bf7073-efea-42d3-83eb-e33d2c9137c1 [e2e-llm-inference-service] resourceVersion: '63549' [e2e-llm-inference-service] uid: cc9154d3-1fbd-4958-a84d-1419eff0f892 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-epp-service.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-epp-service.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:13:10Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"c0bf7073-efea-42d3-83eb-e33d2c9137c1"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T18:13:10Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-shadow-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: c0bf7073-efea-42d3-83eb-e33d2c9137c1 [e2e-llm-inference-service] resourceVersion: '63755' [e2e-llm-inference-service] uid: 1d31126e-5b23-43d8-ad63-5094dd419234 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-inference--ip-8bbb5aea.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:13:02Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"c0bf7073-efea-42d3-83eb-e33d2c9137c1"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T18:13:02Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: c0bf7073-efea-42d3-83eb-e33d2c9137c1 [e2e-llm-inference-service] resourceVersion: '63559' [e2e-llm-inference-service] uid: 71e7cf25-874a-4b9e-82d3-4bbb9ee82d5c [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:13:02Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"c0bf7073-efea-42d3-83eb-e33d2c9137c1"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T18:13:02Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-scheduler [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: c0bf7073-efea-42d3-83eb-e33d2c9137c1 [e2e-llm-inference-service] resourceVersion: '63549' [e2e-llm-inference-service] uid: cc9154d3-1fbd-4958-a84d-1419eff0f892 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-epp-service.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-epp-service.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:13:10Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"c0bf7073-efea-42d3-83eb-e33d2c9137c1"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T18:13:10Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-shadow-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: c0bf7073-efea-42d3-83eb-e33d2c9137c1 [e2e-llm-inference-service] resourceVersion: '63755' [e2e-llm-inference-service] uid: 1d31126e-5b23-43d8-ad63-5094dd419234 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-inference--ip-8bbb5aea.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:13:02Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"c0bf7073-efea-42d3-83eb-e33d2c9137c1"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T18:13:02Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: c0bf7073-efea-42d3-83eb-e33d2c9137c1 [e2e-llm-inference-service] resourceVersion: '63559' [e2e-llm-inference-service] uid: 71e7cf25-874a-4b9e-82d3-4bbb9ee82d5c [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:13:02Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"c0bf7073-efea-42d3-83eb-e33d2c9137c1"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T18:13:02Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-scheduler [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: c0bf7073-efea-42d3-83eb-e33d2c9137c1 [e2e-llm-inference-service] resourceVersion: '63549' [e2e-llm-inference-service] uid: cc9154d3-1fbd-4958-a84d-1419eff0f892 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-epp-service.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-epp-service.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:13:10Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"c0bf7073-efea-42d3-83eb-e33d2c9137c1"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T18:13:10Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-shadow-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: c0bf7073-efea-42d3-83eb-e33d2c9137c1 [e2e-llm-inference-service] resourceVersion: '63755' [e2e-llm-inference-service] uid: 1d31126e-5b23-43d8-ad63-5094dd419234 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-inference--ip-8bbb5aea.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:13:02Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"c0bf7073-efea-42d3-83eb-e33d2c9137c1"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T18:13:02Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: c0bf7073-efea-42d3-83eb-e33d2c9137c1 [e2e-llm-inference-service] resourceVersion: '63559' [e2e-llm-inference-service] uid: 71e7cf25-874a-4b9e-82d3-4bbb9ee82d5c [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"c0bf7073-efea-42d3-83eb-e33d2c9137c1"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:extensionRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:portNumber: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPortNumber: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T18:13:01Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: c0bf7073-efea-42d3-83eb-e33d2c9137c1 [e2e-llm-inference-service] resourceVersion: '63524' [e2e-llm-inference-service] uid: 4a504612-6966-47b1-ad47-269b4d1dadd7 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] extensionRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-service [e2e-llm-inference-service] portNumber: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPortNumber: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parent: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '1970-01-01T00:00:00Z' [e2e-llm-inference-service] message: Waiting for controller [e2e-llm-inference-service] reason: Pending [e2e-llm-inference-service] status: Unknown [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Status [e2e-llm-inference-service] name: default [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:27:49Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 7b89cd7957 [e2e-llm-inference-service] timestamp: '2026-07-28T18:27:42Z' [e2e-llm-inference-service] window: 16.629s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 95692344n [e2e-llm-inference-service] memory: 2407268Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche6mt66 [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:27:49Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 7fc69df7b8 [e2e-llm-inference-service] timestamp: '2026-07-28T18:27:42Z' [e2e-llm-inference-service] window: 16.223s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 54969487n [e2e-llm-inference-service] memory: 33224Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_inference_service] [2026-07-28T18:27:49.627280] end - ❌ 902.138s: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T18:13:12Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:13:12Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:13:12Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:15:46Z', 'severity': 'Info', 'status': 'True', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:13:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:13:12Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:13:36Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:13:36Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:13:12Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ---------------------------- Captured log teardown ----------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.conftest:conftest.py:168 Skipping deletion of namespace e2e-test-llm-inference-service-48639af5 (SKIP_DELETION_ON_FAILURE) [e2e-llm-inference-service] _ test_llm_autoscaling_hpa_lws[router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-hpa] _ [e2e-llm-inference-service] [gw0] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-lws', 'prometheus-scrape', 'scaling-hpa'], prompt='KSer... {'name': 'scaling-hpa-autoscale-hpa-lws-b344a3ff'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.autoscaling_hpa [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator-lws", [e2e-llm-inference-service] "prometheus-scrape", [e2e-llm-inference-service] "scaling-hpa", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="autoscale-hpa-lws", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_multi_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_autoscaling_hpa_lws(test_case: TestCase): [e2e-llm-inference-service] """HPA + LWS: HPA exists with WVA annotations; pods scale under load.""" [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] kserve_client = _new_kserve_client() [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] ns = test_case.namespace [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > _create_and_wait(kserve_client, test_case) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:670: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-lws', 'prometheus-scrape', 'scaling-hpa'], prompt='KSer... {'name': 'scaling-hpa-autoscale-hpa-lws-b344a3ff'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] def _create_and_wait(kserve_client, test_case): [e2e-llm-inference-service] """Create LLMISVC and wait for it to be ready.""" [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:482: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...ale-hpa-b29acdba'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-lws-b344a3ff'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-07-28T18:23:46.740455', start_time = 1785263026.7408257 [e2e-llm-inference-service] duration = 900.0972309112549, timestamp_end = '2026-07-28T18:38:46.838059' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....autoscale-hpa-b29acdba'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-lws-b344a3ff'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1376: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7f67b9016660> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1387: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T18:24:28Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:24:28Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:24:01Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:24:39Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:24:39Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:24:28Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:24:39Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:24:28Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:24:28Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1371: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:82 Created test namespace e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret seaweedfs-s3-creds from kserve-ci-e2e-test to e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret storage-config from kserve-ci-e2e-test to e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:178 Patched default SA in e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 with secret seaweedfs-s3-creds [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:162 ConfigMap odh-kserve-custom-ca-bundle already exists in e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:159 Copied ConfigMap odh-trusted-ca-bundle from kserve-ci-e2e-test to e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig router-managed-autoscale-hpa-lw-1aa98714 in namespace e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig router-managed-autoscale-hpa-lw-1aa98714 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig router-managed-autoscale-hpa-lw-1aa98714 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig workload-llmd-simulator-lws-aut-fe7a55cc in namespace e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig workload-llmd-simulator-lws-aut-fe7a55cc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig workload-llmd-simulator-lws-aut-fe7a55cc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig prometheus-scrape-autoscale-hpa-b29acdba in namespace e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig prometheus-scrape-autoscale-hpa-b29acdba [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig prometheus-scrape-autoscale-hpa-b29acdba [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig scaling-hpa-autoscale-hpa-lws-b344a3ff in namespace e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig scaling-hpa-autoscale-hpa-lws-b344a3ff [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig scaling-hpa-autoscale-hpa-lws-b344a3ff [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_autoscaling_hpa_lws] [2026-07-28T18:23:46.402986] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-llmd-simulator-lws', 'prometheus-scrape', 'scaling-hpa'], prompt='KServe is a', service_name='autoscale-hpa-lws', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, namespace='e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2', before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-lws', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-lw-1aa98714'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-lws-aut-fe7a55cc'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-b29acdba'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-lws-b344a3ff'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-28T18:23:46.415647] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-lws', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-lw-1aa98714'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-lws-aut-fe7a55cc'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-b29acdba'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-lws-b344a3ff'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-28T18:23:46.740310] end - ✅ in 0.324s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-28T18:23:46.740455] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-lws', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-lw-1aa98714'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-lws-aut-fe7a55cc'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-b29acdba'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-lws-b344a3ff'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'RouterReady', 'WorkloadsReady'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T18:24:01Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:24:01Z', 'message': 'Inference Pool e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:24:01Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:24:01Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:24:01Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:24:01Z', 'message': 'HPA not yet visible in cache', 'reason': 'HPAProgressing', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:24:01Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:24:01Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:24:01Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'RouterReady', 'WorkloadsReady'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T18:24:28Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:24:28Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:24:01Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:24:28Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:24:28Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:24:28Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:24:28Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:24:28Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:24:28Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T18:24:28Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:24:28Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:24:01Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:24:39Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:24:39Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:24:28Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:24:39Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:24:28Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:24:28Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 11: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-28T17:29:27Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 12: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-28T17:29:27Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 13: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-28T17:29:27Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1391 Timed out waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T18:24:28Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:24:28Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:24:01Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:24:39Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:24:39Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:24:28Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:24:39Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:24:28Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:24:28Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-07-28T18:38:46.838059] end - ❌ 900.097s: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T18:24:28Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:24:28Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:24:01Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:24:39Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:24:39Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:24:28Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:24:39Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:24:28Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:24:28Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [delete_llmisvc] [2026-07-28T18:38:46.838279] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-lws', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-lw-1aa98714'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-lws-aut-fe7a55cc'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-b29acdba'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-lws-b344a3ff'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 3 pod(s) for autoscale-hpa-lws still terminating: ['autoscale-hpa-lws-kserve-mn-0', 'autoscale-hpa-lws-kserve-mn-0-1', 'autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-sr86h'] [e2e-llm-inference-service] assert not ['autoscale-hpa-lws-kserve-mn-0', 'autoscale-hpa-lws-kserve-mn-0-1', 'autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-sr86h'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 1 pod(s) for autoscale-hpa-lws still terminating: ['autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-sr86h'] [e2e-llm-inference-service] assert not ['autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-sr86h'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [delete_llmisvc] [2026-07-28T18:39:32.246416] end - ✅ in 45.408s [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_autoscaling_hpa_lws] [2026-07-28T18:39:32.246549] end - ❌ 945.843s: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T18:24:28Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:24:28Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:24:01Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:24:39Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:24:39Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:24:28Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:24:39Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:24:28Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:24:28Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ---------------------------- Captured log teardown ----------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.conftest:conftest.py:168 Skipping deletion of namespace e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 (SKIP_DELETION_ON_FAILURE) [e2e-llm-inference-service] _ test_llm_autoscaling_keda_lws[router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-keda] _ [e2e-llm-inference-service] [gw0] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-lws', 'prometheus-scrape', 'scaling-keda'], prompt='KSe... {'name': 'scaling-keda-autoscale-keda-lws-1337f511'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.autoscaling_keda [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator-lws", [e2e-llm-inference-service] "prometheus-scrape", [e2e-llm-inference-service] "scaling-keda", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="autoscale-keda-lws", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_multi_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_autoscaling_keda_lws(test_case: TestCase): [e2e-llm-inference-service] """KEDA + LWS: ScaledObject exists with WVA annotations; pods scale under load.""" [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] kserve_client = _new_kserve_client() [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] ns = test_case.namespace [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > _create_and_wait(kserve_client, test_case) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:728: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-lws', 'prometheus-scrape', 'scaling-keda'], prompt='KSe... {'name': 'scaling-keda-autoscale-keda-lws-1337f511'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] def _create_and_wait(kserve_client, test_case): [e2e-llm-inference-service] """Create LLMISVC and wait for it to be ready.""" [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:482: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...e-ked-231d315d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-lws-1337f511'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-07-28T18:39:33.837263', start_time = 1785263973.8376584 [e2e-llm-inference-service] duration = 900.1278836727142, timestamp_end = '2026-07-28T18:54:33.965545' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....toscale-ked-231d315d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-lws-1337f511'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1376: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7f67b9016480> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1387: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T18:40:27Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:40:27Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:40:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:40:35Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:40:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:40:27Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:40:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:40:27Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:40:27Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1371: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:82 Created test namespace e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret seaweedfs-s3-creds from kserve-ci-e2e-test to e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret storage-config from kserve-ci-e2e-test to e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:178 Patched default SA in e2e-test-llm-autoscaling-keda-lws-e541a132 with secret seaweedfs-s3-creds [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:162 ConfigMap odh-kserve-custom-ca-bundle already exists in e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:159 Copied ConfigMap odh-trusted-ca-bundle from kserve-ci-e2e-test to e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig router-managed-autoscale-keda-l-78828c4a in namespace e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig router-managed-autoscale-keda-l-78828c4a [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig router-managed-autoscale-keda-l-78828c4a [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig workload-llmd-simulator-lws-aut-1696d0b7 in namespace e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig workload-llmd-simulator-lws-aut-1696d0b7 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig workload-llmd-simulator-lws-aut-1696d0b7 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig prometheus-scrape-autoscale-ked-231d315d in namespace e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig prometheus-scrape-autoscale-ked-231d315d [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig prometheus-scrape-autoscale-ked-231d315d [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig scaling-keda-autoscale-keda-lws-1337f511 in namespace e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig scaling-keda-autoscale-keda-lws-1337f511 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig scaling-keda-autoscale-keda-lws-1337f511 [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_autoscaling_keda_lws] [2026-07-28T18:39:33.376541] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-llmd-simulator-lws', 'prometheus-scrape', 'scaling-keda'], prompt='KServe is a', service_name='autoscale-keda-lws', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, namespace='e2e-test-llm-autoscaling-keda-lws-e541a132', before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-keda-lws', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-keda-lws-e541a132', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-keda-l-78828c4a'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-lws-aut-1696d0b7'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-ked-231d315d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-lws-1337f511'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-28T18:39:33.389299] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-keda-lws', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-keda-lws-e541a132', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-keda-l-78828c4a'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-lws-aut-1696d0b7'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-ked-231d315d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-lws-1337f511'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-28T18:39:33.837060] end - ✅ in 0.448s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-28T18:39:33.837263] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-keda-lws', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-keda-lws-e541a132', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-keda-l-78828c4a'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-lws-aut-1696d0b7'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-ked-231d315d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-lws-1337f511'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: No conditions found in status [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 14: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-28T17:29:27Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'RouterReady', 'WorkloadsReady'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T18:40:03Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:40:03Z', 'message': 'Inference Pool e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:40:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:40:03Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:40:03Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:40:03Z', 'message': 'ScaledObject conditions not yet available', 'reason': 'ScaledObjectProgressing', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:40:03Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:40:03Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:40:03Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'RouterReady', 'WorkloadsReady'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T18:40:27Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:40:27Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:40:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:40:27Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:40:27Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:40:27Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:40:27Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:40:27Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:40:27Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T18:40:27Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:40:27Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:40:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:40:35Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:40:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:40:27Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:40:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:40:27Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:40:27Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 15: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-28T17:29:27Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 16: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-28T17:29:27Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1391 Timed out waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T18:40:27Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:40:27Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:40:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:40:35Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:40:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:40:27Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:40:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:40:27Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:40:27Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-07-28T18:54:33.965545] end - ❌ 900.128s: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T18:40:27Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:40:27Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:40:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:40:35Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:40:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:40:27Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:40:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:40:27Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:40:27Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [delete_llmisvc] [2026-07-28T18:54:33.965627] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-keda-lws', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-keda-lws-e541a132', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-keda-l-78828c4a'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-lws-aut-1696d0b7'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-ked-231d315d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-lws-1337f511'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 3 pod(s) for autoscale-keda-lws still terminating: ['autoscale-keda-lws-kserve-mn-0', 'autoscale-keda-lws-kserve-mn-0-1', 'autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-sv5qr'] [e2e-llm-inference-service] assert not ['autoscale-keda-lws-kserve-mn-0', 'autoscale-keda-lws-kserve-mn-0-1', 'autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-sv5qr'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 1 pod(s) for autoscale-keda-lws still terminating: ['autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-sv5qr'] [e2e-llm-inference-service] assert not ['autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-sv5qr'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [delete_llmisvc] [2026-07-28T18:54:54.331431] end - ✅ in 20.366s [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_autoscaling_keda_lws] [2026-07-28T18:54:54.331588] end - ❌ 920.955s: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T18:40:27Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:40:27Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:40:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:40:35Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:40:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:40:27Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:40:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:40:27Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:40:27Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ---------------------------- Captured log teardown ----------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.conftest:conftest.py:168 Skipping deletion of namespace e2e-test-llm-autoscaling-keda-lws-e541a132 (SKIP_DELETION_ON_FAILURE) [e2e-llm-inference-service] =============================== warnings summary =============================== [e2e-llm-inference-service] llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-utilization-detector] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] /workspace/source/python/kserve/.venv/lib64/python3.11/site-packages/pytest_asyncio/plugin.py:761: DeprecationWarning: The event_loop fixture provided by pytest-asyncio has been redefined in [e2e-llm-inference-service] /workspace/source/test/e2e/conftest.py:43 [e2e-llm-inference-service] Replacing the event_loop fixture with a custom implementation is deprecated [e2e-llm-inference-service] and will lead to errors in the future. [e2e-llm-inference-service] If you want to request an asyncio event loop with a scope other than function [e2e-llm-inference-service] scope, use the "scope" argument to the asyncio mark when marking the tests. [e2e-llm-inference-service] If you want to return different types of event loops, use the event_loop_policy [e2e-llm-inference-service] fixture. [e2e-llm-inference-service] [e2e-llm-inference-service] warnings.warn( [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-utilization-detector] [e2e-llm-inference-service] llmisvc/test_flow_control.py:47: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-concurrency-detector] [e2e-llm-inference-service] llmisvc/test_flow_control.py:47: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-inline-config-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-qwen2.5-0.5b] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-configmap-ref-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-replicas-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-custom-template-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-nonzero-threshold-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-tokenizer-kvcache-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator2] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service_stop.py::test_llm_stop_feature[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service_stop.py:40: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_tls.py::test_llm_tls_resources[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_tls.py:92: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html [e2e-llm-inference-service] - generated xml file: /workspace/artifacts-dir/junit_e2e-llminferenceservice_and_cluster_cpu_and_not_autoscaling_and_not_tracing.xml - [e2e-llm-inference-service] --------------------------------- JSON report ---------------------------------- [e2e-llm-inference-service] report saved to: /workspace/artifacts-dir/e2e_results-llminferenceservice_and_cluster_cpu_and_not_autoscaling_and_not_tracing.json [e2e-llm-inference-service] =========================== short test summary info ============================ [e2e-llm-inference-service] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] FAILED llmisvc/test_llm_auth.py::test_llm_auth_enabled_requires_token[cluster_cpu-cluster_single_node-auth-enabled-default] [e2e-llm-inference-service] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_hpa_deployment[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_keda_deployment[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-keda] [e2e-llm-inference-service] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-pvc] [e2e-llm-inference-service] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_hpa_lws[cluster_cpu-cluster_multi_node-router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_keda_lws[cluster_cpu-cluster_multi_node-router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-keda] [e2e-llm-inference-service] !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 7 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! [e2e-llm-inference-service] !!!!!!!!!!!! xdist.dsession.Interrupted: stopping after 5 failures !!!!!!!!!!!!! [e2e-llm-inference-service] ====== 7 failed, 41 passed, 3 skipped, 25 warnings in 5150.84s (1:25:50) ======= [must-gather] [must-gather ] OUT 2026-07-28T18:54:59.582268571Z Using must-gather plug-in image: quay.io/modh/must-gather:rhoai-2.24 [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: 331d3e0a-05b1-4db5-9f55-68440dff074b [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.26" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [must-gather] [must-gather ] OUT 2026-07-28T18:54:59.627266557Z namespace/openshift-must-gather-tpd97 created [must-gather] [must-gather ] OUT 2026-07-28T18:54:59.635278341Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-6ctm6 created [must-gather] [must-gather ] OUT 2026-07-28T18:54:59.664446842Z pod for plug-in image quay.io/modh/must-gather:rhoai-2.24 created [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:06.449153859Z [disk usage checker] Started [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:06.452625055Z [disk usage checker] Volume usage percentage: current = 21 ; allowed = 70 [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:06.718200071Z Error from server (NotFound): namespaces "redhat-ods-operator" not found [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:06.882239842Z Wrote inspect data to must-gather. [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:06.882274259Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:06.882274259Z namespaces "redhat-ods-operator" not found [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:06.884787434Z Error getting logs from redhat-ods-operator [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:07.056534589Z Wrote inspect data to must-gather. [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:07.056565551Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:07.056565551Z namespaces "redhat-ods-monitoring" not found [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:07.058848876Z Error getting logs from redhat-ods-monitoring [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:07.225608924Z Wrote inspect data to must-gather. [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:07.225640635Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:07.225640635Z namespaces "redhat-ods-applications" not found [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:07.228040887Z Error getting logs from redhat-ods-applications [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:07.400795816Z Wrote inspect data to must-gather. [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:07.400829274Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:07.400829274Z namespaces "rhods-notebooks" not found [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:07.403334181Z Error getting logs from rhods-notebooks [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:07.569192065Z Wrote inspect data to must-gather. [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:07.569224300Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:07.569224300Z namespaces "rhoai-model-registries" not found [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:07.571209778Z Error getting logs from rhoai-model-registries [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:07.739815359Z Wrote inspect data to must-gather. [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:07.739847541Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:07.739847541Z namespaces "istio-system" not found [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:07.742329664Z Error getting logs from istio-system [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:07.917766058Z Wrote inspect data to must-gather. [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:07.917799732Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:07.917799732Z namespaces "knative-serving" not found [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:07.920647171Z Error getting logs from knative-serving [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:08.095730785Z Wrote inspect data to must-gather. [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:08.095762552Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:08.095762552Z namespaces "redhat-ods-applications-auth-provider" not found [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:08.098231090Z Error getting logs from redhat-ods-applications-auth-provider [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:08.400904850Z Wrote inspect data to must-gather. [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:08.700658340Z Wrote inspect data to must-gather. [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:08.847331841Z error: the server doesn't have a resource type "auths" [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:08.932828656Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:08.935891929Z Error collecting info from [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:09.073857685Z error: the server doesn't have a resource type "monitorings" [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:09.160821044Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:09.162965237Z Error collecting info from [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:09.304586654Z error: the server doesn't have a resource type "featuretrackers" [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:09.394095428Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:09.396261189Z Error collecting info from [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:09.533371590Z error: the server doesn't have a resource type "codeflares" [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:09.620834330Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:09.623003500Z Error collecting info from [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:09.759247383Z error: the server doesn't have a resource type "dashboards" [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:09.852840445Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:09.855460529Z Error collecting info from [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:09.995310280Z error: the server doesn't have a resource type "datasciencepipelines" [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:10.084692660Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:10.087137069Z Error collecting info from [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:10.231801020Z error: the server doesn't have a resource type "feastoperators" [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:10.320486368Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:10.322889643Z Error collecting info from [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:10.463355914Z error: the server doesn't have a resource type "kserves" [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:10.552827476Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:10.555295736Z Error collecting info from [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:10.692729689Z error: the server doesn't have a resource type "kueues" [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:10.780506674Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:10.782813995Z Error collecting info from [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:10.925982009Z error: the server doesn't have a resource type "modelcontrollers" [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:11.010543845Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:11.012904074Z Error collecting info from [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:11.150285883Z error: the server doesn't have a resource type "modelmeshservings" [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:11.240182652Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:11.242333928Z Error collecting info from [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:11.390385600Z error: the server doesn't have a resource type "modelregistries" [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:11.457349644Z [disk usage checker] Volume usage percentage: current = 21 ; allowed = 70 [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:11.477359398Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:11.479508364Z Error collecting info from [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:11.616820292Z error: the server doesn't have a resource type "rays" [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:11.701838464Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:11.703896121Z Error collecting info from [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:11.842772303Z error: the server doesn't have a resource type "trainingoperators" [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:11.933371672Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:11.935980113Z Error collecting info from [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:12.071343803Z error: the server doesn't have a resource type "trustyais" [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:12.160660712Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:12.162714842Z Error collecting info from [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:12.306563953Z error: the server doesn't have a resource type "workbenches" [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:12.392171590Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:12.394325699Z Error collecting info from [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:12.530509312Z error: the server doesn't have a resource type "hardwareprofiles" [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:12.615570117Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:12.617872336Z Error collecting info from [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:12.760604705Z error: the server doesn't have a resource type "llamastackoperators" [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:12.848579125Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:12.851364920Z Error collecting info from [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:13.598963127Z error: the server doesn't have a resource type "predictors" [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:13.734873021Z error: the server doesn't have a resource type "localmodelnodegroups" [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:13.870998932Z error: the server doesn't have a resource type "smcp" [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:14.006719413Z error: the server doesn't have a resource type "smm" [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:14.135879675Z error: the server doesn't have a resource type "smmr" [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:14.742441216Z error: the server doesn't have a resource type "knativeservings" [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:14.882389714Z error: the server doesn't have a resource type "configurations" [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:15.026492058Z error: the server doesn't have a resource type "routes" [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:15.165241875Z error: the server doesn't have a resource type "services" [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:15.300163903Z error: the server doesn't have a resource type "revisions" [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:15.890399627Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:16.028505144Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:16.121355066Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:16.200400774Z Wrote inspect data to must-gather. [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:16.367784602Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:16.462947932Z [disk usage checker] Volume usage percentage: current = 21 ; allowed = 70 [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:16.505185904Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:16.604994285Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:16.686905579Z Wrote inspect data to must-gather. [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:16.851083936Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:16.981550588Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:17.065502376Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:17.139347780Z Wrote inspect data to must-gather. [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:17.299264239Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:17.433985299Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:17.513668660Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:17.589116265Z Wrote inspect data to must-gather. [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:17.751897479Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:17.887909508Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:17.972799046Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:18.051842398Z Wrote inspect data to must-gather. [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:18.219383273Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:18.367226150Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:18.453776931Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:18.530060767Z Wrote inspect data to must-gather. [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:18.694982697Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:18.909323423Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:18.997666615Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:19.075928304Z Wrote inspect data to must-gather. [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:19.251248593Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:19.397536222Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:19.486027318Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:19.563563031Z Wrote inspect data to must-gather. [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:19.733942051Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:19.883743671Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:19.971763521Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:20.051563111Z Wrote inspect data to must-gather. [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:20.218147957Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:20.360521397Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:20.447958357Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:20.531798758Z Wrote inspect data to must-gather. [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:20.699582688Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:20.836053314Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:20.917854752Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:20.991893289Z Wrote inspect data to must-gather. [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:21.169477288Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:21.313266108Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:21.397699235Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:21.469362905Z [disk usage checker] Volume usage percentage: current = 21 ; allowed = 70 [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:21.477369217Z Wrote inspect data to must-gather. [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:21.657230773Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:21.803511023Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:21.891410740Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:21.971342301Z Wrote inspect data to must-gather. [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:22.136162966Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:22.275272604Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:22.365124362Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:22.441637337Z Wrote inspect data to must-gather. [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:22.614994641Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:23.489491869Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:23.658292837Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:23.770005886Z Wrote inspect data to must-gather. [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:23.941654937Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:24.871833354Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:25.204043543Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:25.377880709Z Wrote inspect data to must-gather. [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:25.554118845Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:26.474035262Z [disk usage checker] Volume usage percentage: current = 21 ; allowed = 70 [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:26.480026440Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:26.730455785Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:26.848259891Z Wrote inspect data to must-gather. [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:27.028399481Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:27.842127369Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:28.005198669Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:28.123752213Z Wrote inspect data to must-gather. [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:28.298481735Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:28.974485298Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:29.178543558Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:29.301872214Z Wrote inspect data to must-gather. [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:29.475093034Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:30.169499164Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:30.361371437Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:30.487325445Z Wrote inspect data to must-gather. [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:30.654857317Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:31.330550087Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:31.478459195Z [disk usage checker] Volume usage percentage: current = 21 ; allowed = 70 [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:31.514315283Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:31.638996965Z Wrote inspect data to must-gather. [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:31.806136044Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:32.469300994Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:32.656147586Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:32.783592291Z Wrote inspect data to must-gather. [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:32.949937695Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:33.620684505Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:33.807124019Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:33.932505356Z Wrote inspect data to must-gather. [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:34.101359522Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:34.775468058Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:34.966489323Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:35.091557015Z Wrote inspect data to must-gather. [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:35.261612479Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:35.941218586Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:36.142377364Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:36.265156091Z Wrote inspect data to must-gather. [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:36.436127534Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:36.483117715Z [disk usage checker] Volume usage percentage: current = 21 ; allowed = 70 [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:37.112249485Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:37.304157548Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:37.427834388Z Wrote inspect data to must-gather. [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:37.594598689Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:38.273451172Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:38.475803375Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:38.600797143Z Wrote inspect data to must-gather. [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:38.770908570Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:39.466414574Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:39.682420641Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:39.803411716Z Wrote inspect data to must-gather. [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:39.972419293Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:40.625483900Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:40.810327580Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:40.952461170Z Wrote inspect data to must-gather. [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:41.120234360Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:41.488971467Z [disk usage checker] Volume usage percentage: current = 21 ; allowed = 70 [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:41.796416048Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:41.990674886Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:42.110370153Z Wrote inspect data to must-gather. [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:42.283848942Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:42.953188725Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:43.142627451Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:43.266650960Z Wrote inspect data to must-gather. [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:43.436317054Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:44.115800845Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:44.305745506Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:44.428159282Z Wrote inspect data to must-gather. [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:44.593431938Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:45.269062151Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:45.456986637Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:45.580406466Z Wrote inspect data to must-gather. [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:45.752986079Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:46.423712248Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:46.493790798Z [disk usage checker] Volume usage percentage: current = 21 ; allowed = 70 [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:46.613312319Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:46.737560607Z Wrote inspect data to must-gather. [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:46.910123012Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:47.562348161Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:47.745699145Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:47.868799364Z Wrote inspect data to must-gather. [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:48.044434148Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:48.690578115Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:48.875630008Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:48.993947776Z Wrote inspect data to must-gather. [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:49.167470391Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:49.849342314Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:50.050354197Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:50.192882536Z Wrote inspect data to must-gather. [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:50.368305987Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:51.065178581Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:51.256342939Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:51.379318840Z Wrote inspect data to must-gather. [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:51.498248943Z [disk usage checker] Volume usage percentage: current = 21 ; allowed = 70 [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:51.552936461Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:52.226500535Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:52.413962210Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:52.535941513Z Wrote inspect data to must-gather. [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:52.703834272Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:53.371929349Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:53.554925006Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:53.678926199Z Wrote inspect data to must-gather. [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:53.846892675Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:54.519790417Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:54.706852362Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:54.833302798Z Wrote inspect data to must-gather. [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:55.010690528Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:55.682110551Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:55.870767811Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:55.991441714Z Wrote inspect data to must-gather. [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:56.158408887Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:56.502745480Z [disk usage checker] Volume usage percentage: current = 21 ; allowed = 70 [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:56.817251110Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:57.004501881Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:57.125956672Z Wrote inspect data to must-gather. [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:57.293589498Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:57.950697441Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:58.133954414Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:58.254616425Z Wrote inspect data to must-gather. [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:58.425135676Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:59.099116503Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:59.285199935Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:59.403833868Z Wrote inspect data to must-gather. [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:55:59.570116526Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:56:00.263772528Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:56:00.472947532Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:56:00.597844219Z Wrote inspect data to must-gather. [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:56:00.770855238Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:56:01.441620346Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:56:01.507300260Z [disk usage checker] Volume usage percentage: current = 21 ; allowed = 70 [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:56:01.630099709Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:56:01.758039279Z Wrote inspect data to must-gather. [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:56:01.928036745Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:56:02.581161715Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:56:02.769198570Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:56:02.888118202Z Wrote inspect data to must-gather. [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:56:03.056509353Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:56:03.749655087Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:56:03.948515272Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:56:04.071251154Z Wrote inspect data to must-gather. [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:56:04.238486625Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:56:04.893614667Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:56:05.114365706Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:56:05.253957563Z Wrote inspect data to must-gather. [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:56:05.425231140Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:56:06.093193210Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:56:06.283375933Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:56:06.406197200Z Wrote inspect data to must-gather. [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:56:06.511761642Z [disk usage checker] Volume usage percentage: current = 21 ; allowed = 70 [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:56:06.575875221Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:56:07.236446012Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:56:07.424765794Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:56:07.546564464Z Wrote inspect data to must-gather. [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:56:07.711126557Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:56:08.358280136Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:56:08.541019991Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:56:08.663454340Z Wrote inspect data to must-gather. [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:56:08.830569242Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:56:09.494532598Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:56:09.684306399Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:56:09.810588997Z Wrote inspect data to must-gather. [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:56:09.975549155Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:56:10.623197613Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:56:10.816883278Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:56:10.962351500Z Wrote inspect data to must-gather. [must-gather] [must-gather-k4ppp] POD 2026-07-28T18:56:10.986106718Z Caches written to disk [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:11.997478086Z waiting for gather to complete [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.002734136Z downloading gather output [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.30006958Z receiving incremental file list [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.315420161Z ./ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.315468022Z aggregated-discovery-api.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.315799289Z aggregated-discovery-apis.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.316763269Z event-filter.html [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.317151637Z timestamp [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.31729354Z version [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.321978658Z cluster-scoped-resources/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.321997118Z cluster-scoped-resources/datasciencecluster.opendatahub.io/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.322006218Z cluster-scoped-resources/datasciencecluster.opendatahub.io/datascienceclusters/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.322048289Z cluster-scoped-resources/datasciencecluster.opendatahub.io/datascienceclusters/test-dsc.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.322156291Z cluster-scoped-resources/dscinitialization.opendatahub.io/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.322178912Z cluster-scoped-resources/dscinitialization.opendatahub.io/dscinitializations/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.322229803Z cluster-scoped-resources/dscinitialization.opendatahub.io/dscinitializations/test-dsci.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.322304814Z namespaces/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.322317554Z namespaces/kserve-ci-e2e-test/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.322483728Z namespaces/kserve-ci-e2e-test/kserve-ci-e2e-test.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.3225715Z namespaces/kserve-ci-e2e-test/apps.openshift.io/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.322643331Z namespaces/kserve-ci-e2e-test/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.322708543Z namespaces/kserve-ci-e2e-test/apps/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.322759834Z namespaces/kserve-ci-e2e-test/apps/daemonsets.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.322888957Z namespaces/kserve-ci-e2e-test/apps/deployments.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.323004709Z namespaces/kserve-ci-e2e-test/apps/replicasets.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.323116391Z namespaces/kserve-ci-e2e-test/apps/statefulsets.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.323200273Z namespaces/kserve-ci-e2e-test/autoscaling/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.323257304Z namespaces/kserve-ci-e2e-test/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.323316725Z namespaces/kserve-ci-e2e-test/batch/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.323365326Z namespaces/kserve-ci-e2e-test/batch/cronjobs.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.323491369Z namespaces/kserve-ci-e2e-test/batch/jobs.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.3235558Z namespaces/kserve-ci-e2e-test/build.openshift.io/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.323620832Z namespaces/kserve-ci-e2e-test/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.323741094Z namespaces/kserve-ci-e2e-test/build.openshift.io/builds.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.323807176Z namespaces/kserve-ci-e2e-test/core/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.323856507Z namespaces/kserve-ci-e2e-test/core/configmaps.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.324055141Z namespaces/kserve-ci-e2e-test/core/endpoints.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.324190964Z namespaces/kserve-ci-e2e-test/core/events.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.324320056Z namespaces/kserve-ci-e2e-test/core/persistentvolumeclaims.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.324430599Z namespaces/kserve-ci-e2e-test/core/pods.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.324548891Z namespaces/kserve-ci-e2e-test/core/replicationcontrollers.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.324714704Z namespaces/kserve-ci-e2e-test/core/secrets.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.32497677Z namespaces/kserve-ci-e2e-test/core/services.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.325050422Z namespaces/kserve-ci-e2e-test/discovery.k8s.io/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.325107103Z namespaces/kserve-ci-e2e-test/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.325237306Z namespaces/kserve-ci-e2e-test/image.openshift.io/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.325305917Z namespaces/kserve-ci-e2e-test/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.325371138Z namespaces/kserve-ci-e2e-test/k8s.ovn.org/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.32544898Z namespaces/kserve-ci-e2e-test/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.325562862Z namespaces/kserve-ci-e2e-test/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.325623114Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.325688545Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.325755066Z namespaces/kserve-ci-e2e-test/networking.k8s.io/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.325822758Z namespaces/kserve-ci-e2e-test/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.325889419Z namespaces/kserve-ci-e2e-test/policy/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.325975481Z namespaces/kserve-ci-e2e-test/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.326048552Z namespaces/kserve-ci-e2e-test/route.openshift.io/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.326145624Z namespaces/kserve-ci-e2e-test/route.openshift.io/routes.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.326259507Z namespaces/kuadrant-system/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.326319708Z namespaces/kuadrant-system/kuadrant-system.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.32639159Z namespaces/kuadrant-system/apps.openshift.io/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.326456061Z namespaces/kuadrant-system/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.326512082Z namespaces/kuadrant-system/apps/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.326583574Z namespaces/kuadrant-system/apps/daemonsets.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.326665565Z namespaces/kuadrant-system/apps/deployments.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.327015883Z namespaces/kuadrant-system/apps/replicasets.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.32739265Z namespaces/kuadrant-system/apps/statefulsets.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.327446101Z namespaces/kuadrant-system/autoscaling/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.327488733Z namespaces/kuadrant-system/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.327555954Z namespaces/kuadrant-system/batch/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.327610155Z namespaces/kuadrant-system/batch/cronjobs.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.327711647Z namespaces/kuadrant-system/batch/jobs.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.327761228Z namespaces/kuadrant-system/build.openshift.io/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.32782654Z namespaces/kuadrant-system/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.327923902Z namespaces/kuadrant-system/build.openshift.io/builds.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.327953712Z namespaces/kuadrant-system/core/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.328025924Z namespaces/kuadrant-system/core/configmaps.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.328275009Z namespaces/kuadrant-system/core/endpoints.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.328418332Z namespaces/kuadrant-system/core/events.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.329147667Z namespaces/kuadrant-system/core/persistentvolumeclaims.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.329245049Z namespaces/kuadrant-system/core/pods.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.329711989Z namespaces/kuadrant-system/core/replicationcontrollers.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.329852162Z namespaces/kuadrant-system/core/secrets.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.330097747Z namespaces/kuadrant-system/core/services.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.330286191Z namespaces/kuadrant-system/discovery.k8s.io/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.330331672Z namespaces/kuadrant-system/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.330490465Z namespaces/kuadrant-system/image.openshift.io/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.330526506Z namespaces/kuadrant-system/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.330612467Z namespaces/kuadrant-system/k8s.ovn.org/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.330653379Z namespaces/kuadrant-system/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.330770251Z namespaces/kuadrant-system/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.330819122Z namespaces/kuadrant-system/monitoring.coreos.com/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.330889193Z namespaces/kuadrant-system/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.330943734Z namespaces/kuadrant-system/networking.k8s.io/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.331003676Z namespaces/kuadrant-system/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.331042676Z namespaces/kuadrant-system/pods/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.331053667Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-85tfv/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.331118238Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-85tfv/authorino-7cf4765b85-85tfv.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.33122577Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-85tfv/authorino/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.331243131Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-85tfv/authorino/authorino/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.331250001Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-85tfv/authorino/authorino/logs/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.331305952Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-85tfv/authorino/authorino/logs/current.log [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.343375554Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-85tfv/authorino/authorino/logs/previous.insecure.log [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.343427795Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-85tfv/authorino/authorino/logs/previous.log [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.343474306Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-2kw2j/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.343501116Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-2kw2j/authorino-operator-56fb6f6856-2kw2j.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.343617159Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-2kw2j/manager/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.343631959Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-2kw2j/manager/manager/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.343638049Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-2kw2j/manager/manager/logs/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.34365511Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-2kw2j/manager/manager/logs/current.log [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.343830313Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-2kw2j/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.343903045Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-2kw2j/manager/manager/logs/previous.log [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.343927565Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-qq2zk/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.343994727Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-qq2zk/dns-operator-controller-manager-ccbdf686-qq2zk.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.344087909Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-qq2zk/manager/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.344102209Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-qq2zk/manager/manager/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.344109869Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-qq2zk/manager/manager/logs/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.34414717Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-qq2zk/manager/manager/logs/current.log [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.344256992Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-qq2zk/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.344331584Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-qq2zk/manager/manager/logs/previous.log [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.344344314Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-vrjg8/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.344399735Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-vrjg8/kuadrant-console-plugin-598f86468b-vrjg8.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.344482797Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-vrjg8/kuadrant-console-plugin/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.344491487Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-vrjg8/kuadrant-console-plugin/kuadrant-console-plugin/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.344496607Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-vrjg8/kuadrant-console-plugin/kuadrant-console-plugin/logs/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.344520778Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-vrjg8/kuadrant-console-plugin/kuadrant-console-plugin/logs/current.log [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.34460007Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-vrjg8/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.insecure.log [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.344663131Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-vrjg8/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.log [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.344695292Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-2xg2c/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.344737502Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-2xg2c/kuadrant-operator-controller-manager-66866f5fcc-2xg2c.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.344853115Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-2xg2c/manager/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.344869305Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-2xg2c/manager/manager/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.344875015Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-2xg2c/manager/manager/logs/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.344888205Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-2xg2c/manager/manager/logs/current.log [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.355141939Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-2xg2c/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.355229051Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-2xg2c/manager/manager/logs/previous.log [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.355242632Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-5xp74/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.355304283Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-5xp74/limitador-limitador-85d9db4d5d-5xp74.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.355416665Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-5xp74/limitador/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.355426765Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-5xp74/limitador/limitador/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.355431636Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-5xp74/limitador/limitador/logs/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.355456266Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-5xp74/limitador/limitador/logs/current.log [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.355559638Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-5xp74/limitador/limitador/logs/previous.insecure.log [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.355622409Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-5xp74/limitador/limitador/logs/previous.log [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.35564175Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-xhtkz/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.355701761Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-xhtkz/limitador-operator-controller-manager-774d94476f-xhtkz.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.355793263Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-xhtkz/manager/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.355803803Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-xhtkz/manager/manager/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.355808873Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-xhtkz/manager/manager/logs/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.355819564Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-xhtkz/manager/manager/logs/current.log [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.355987277Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-xhtkz/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.356057538Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-xhtkz/manager/manager/logs/previous.log [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.356069829Z namespaces/kuadrant-system/policy/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.35614531Z namespaces/kuadrant-system/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.356207242Z namespaces/kuadrant-system/route.openshift.io/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.356231332Z namespaces/kuadrant-system/route.openshift.io/routes.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.356290783Z namespaces/openshift-ingress/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.356348104Z namespaces/openshift-ingress/openshift-ingress.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.356394136Z namespaces/openshift-ingress/apps.openshift.io/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.356451287Z namespaces/openshift-ingress/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.356493507Z namespaces/openshift-ingress/apps/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.356532658Z namespaces/openshift-ingress/apps/daemonsets.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.35661428Z namespaces/openshift-ingress/apps/deployments.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.356834275Z namespaces/openshift-ingress/apps/replicasets.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.357035399Z namespaces/openshift-ingress/apps/statefulsets.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.35706875Z namespaces/openshift-ingress/autoscaling/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.357148201Z namespaces/openshift-ingress/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.357213433Z namespaces/openshift-ingress/batch/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.357252193Z namespaces/openshift-ingress/batch/cronjobs.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.357339155Z namespaces/openshift-ingress/batch/jobs.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.357374906Z namespaces/openshift-ingress/build.openshift.io/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.357418367Z namespaces/openshift-ingress/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.357497848Z namespaces/openshift-ingress/build.openshift.io/builds.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.357543689Z namespaces/openshift-ingress/core/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.35759149Z namespaces/openshift-ingress/core/configmaps.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.357983099Z namespaces/openshift-ingress/core/endpoints.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.358079641Z namespaces/openshift-ingress/core/events.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.358442858Z namespaces/openshift-ingress/core/persistentvolumeclaims.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.35852067Z namespaces/openshift-ingress/core/pods.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.358802146Z namespaces/openshift-ingress/core/replicationcontrollers.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.358932829Z namespaces/openshift-ingress/core/secrets.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.359207654Z namespaces/openshift-ingress/core/services.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.359306566Z namespaces/openshift-ingress/discovery.k8s.io/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.359350847Z namespaces/openshift-ingress/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.359430669Z namespaces/openshift-ingress/image.openshift.io/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.359462929Z namespaces/openshift-ingress/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.359515571Z namespaces/openshift-ingress/k8s.ovn.org/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.359548051Z namespaces/openshift-ingress/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.359641453Z namespaces/openshift-ingress/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.359673104Z namespaces/openshift-ingress/monitoring.coreos.com/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.359726565Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.359781976Z namespaces/openshift-ingress/networking.k8s.io/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.359816197Z namespaces/openshift-ingress/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.359857558Z namespaces/openshift-ingress/pods/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.359866498Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-bv8fm/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.359912639Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-bv8fm/istiod-openshift-gateway-5677ccb49b-bv8fm.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.360016231Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-bv8fm/discovery/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.360026501Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-bv8fm/discovery/discovery/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.360031461Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-bv8fm/discovery/discovery/logs/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.360054462Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-bv8fm/discovery/discovery/logs/current.log [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.380041079Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-bv8fm/discovery/discovery/logs/previous.insecure.log [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.38011016Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-bv8fm/discovery/discovery/logs/previous.log [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.380163612Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.380203972Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.380358045Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l/istio-proxy/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.380368116Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l/istio-proxy/istio-proxy/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.380373986Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.380412927Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.3820164Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.382111332Z namespaces/openshift-ingress/pods/router-default-66dc97f78c-hsl5f/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.382172083Z namespaces/openshift-ingress/pods/router-default-66dc97f78c-hsl5f/router-default-66dc97f78c-hsl5f.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.382316196Z namespaces/openshift-ingress/pods/router-default-66dc97f78c-hsl5f/router/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.382326557Z namespaces/openshift-ingress/pods/router-default-66dc97f78c-hsl5f/router/router/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.382331747Z namespaces/openshift-ingress/pods/router-default-66dc97f78c-hsl5f/router/router/logs/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.382366337Z namespaces/openshift-ingress/pods/router-default-66dc97f78c-hsl5f/router/router/logs/current.log [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.38250827Z namespaces/openshift-ingress/pods/router-default-66dc97f78c-hsl5f/router/router/logs/previous.insecure.log [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.382591722Z namespaces/openshift-ingress/pods/router-default-66dc97f78c-hsl5f/router/router/logs/previous.log [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.382638103Z namespaces/openshift-ingress/policy/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.382688514Z namespaces/openshift-ingress/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.382753195Z namespaces/openshift-ingress/route.openshift.io/ [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.382802647Z namespaces/openshift-ingress/route.openshift.io/routes.yaml [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.391564549Z [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.39159295Z sent 2,848 bytes received 1,063,890 bytes 2,133,476.00 bytes/sec [must-gather] [must-gather-k4ppp] OUT 2026-07-28T18:56:12.39161008Z total size is 15,202,470 speedup is 14.25 [must-gather] [must-gather ] OUT 2026-07-28T18:56:12.446316122Z namespace/openshift-must-gather-tpd97 deleted [must-gather] [must-gather] [must-gather] Reprinting Cluster State: [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: 331d3e0a-05b1-4db5-9f55-68440dff074b [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.26" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [must-gather] [must-gather ] OUT 2026-07-28T18:56:12.580020172Z Using must-gather plug-in image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:35f66492587d98a3cca9bf7ba4541c867770dc76b78b4cf9352e6715dca04502 [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: 331d3e0a-05b1-4db5-9f55-68440dff074b [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.26" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [must-gather] [must-gather ] OUT 2026-07-28T18:56:12.606452173Z namespace/openshift-must-gather-ct488 created [must-gather] [must-gather ] OUT 2026-07-28T18:56:12.614227016Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-94tcv created [must-gather] [must-gather ] OUT 2026-07-28T18:56:12.646790025Z pod for plug-in image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:35f66492587d98a3cca9bf7ba4541c867770dc76b78b4cf9352e6715dca04502 created [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:14.035911531Z [disk usage checker] Started [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:14.040010575Z [disk usage checker] Volume usage percentage: current = 21 ; allowed = 70 [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:14.262690866Z Gathering data for ns/openshift-cluster-version... [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:14.417821220Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:14.512539607Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:14.611149534Z Gathering data for ns/default... [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:14.973318240Z Gathering data for ns/openshift... [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:15.446108562Z Gathering data for ns/kube-system... [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:16.264490827Z Gathering data for ns/openshift-etcd... [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:16.632613611Z Wrote inspect data to must-gather. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:16.632646004Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:16.632646004Z namespaces "assisted-installer" not found [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:17.091479909Z Waiting on subprocesses to finish execution. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:17.099285464Z INFO: Gathering HAProxy config files [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:17.101339089Z Executing Istio gather script [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:17.101767185Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:17.101836523Z INFO: Gathering machine config daemon's old logs from all nodes [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:17.107616099Z INFO: Gathering on-disk MachineConfig from degraded nodes [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:17.134987111Z INFO: Collecting host service logs for crio [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:17.135429827Z INFO: Collecting host service logs for kubelet [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:17.135747944Z INFO: Collecting host service logs for rpm-ostreed [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:17.136028298Z INFO: Collecting host service logs for ostree-finalize-staged [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:17.136341992Z INFO: Collecting host service logs for machine-config-daemon-firstboot [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:17.136647687Z INFO: Collecting host service logs for machine-config-daemon-host [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:17.136934774Z INFO: Collecting host service logs for NetworkManager [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:17.137225475Z INFO: Collecting host service logs for openvswitch [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:17.137511723Z INFO: Collecting host service logs for ovs-configuration [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:17.137793818Z INFO: Collecting host service logs for ovsdb-server [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:17.138105302Z INFO: Collecting host service logs for ovs-vswitchd [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:17.138483552Z INFO: Waiting for worker host service log collection to complete ... [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:17.165836338Z INFO: Waiting for node performance related collection to complete ... [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:17.437391148Z INFO: Waiting for on-disk MachineConfig collection to complete ... [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:17.437487033Z INFO: on-disk MachineConfig config collection complete. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:17.607278422Z No resources found in openshift-etcd namespace. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:17.770410085Z INFO: Found 1 replicas - prometheus-k8s-0 [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:18.186961963Z No resources found [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:18.278371940Z INFO: namespace openshift-frr-k8s not detected. Skipping. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:18.324606023Z error: the server doesn't have a resource type "performanceprofile" [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:18.391334501Z INFO: Collecting Insights Archives from insights-operator-5885fdb759-vkqmg [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:18.391381503Z insights-runtime-extractor-bttvz [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:18.391403014Z insights-runtime-extractor-gct9w [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:18.391424288Z insights-runtime-extractor-zp6vz [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:18.508681482Z INFO: "metallb-operator" not detected. Skipping. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:18.610254461Z INFO: "sriov-network-operator" not detected. Skipping. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:18.729267943Z Wrote inspect data to must-gather. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:18.984802865Z INFO: "kubernetes-nmstate-operator" not detected. Skipping. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:19.052918310Z INFO: Worker host service log collection to complete. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:19.070286371Z INFO: Waiting for HAProxy config collection to complete ... [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:19.098661907Z INFO: Getting alertmanagers from prometheus-k8s-0 [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:19.226353832Z INFO: 'previous-logs' folder not found on ip-10-0-135-188.ec2.internal, skipping... [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:19.450176463Z ERROR: No running kube-apiserver pods found [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:19.565886170Z [disk usage checker] Volume usage percentage: current = 21 ; allowed = 70 [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:19.682864352Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:19.704940575Z Wrote inspect data to must-gather. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:19.777648399Z error: the server doesn't have a resource type "multi-networkpolicy" [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:19.871526783Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:19.871526783Z See 'oc rsync -h' for help and examples [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:19.959762549Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:20.253005943Z Wrote inspect data to must-gather. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:20.324612488Z No resources found [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:20.335063352Z INFO: 'previous-logs' folder not found on ip-10-0-135-20.ec2.internal, skipping... [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:20.442278630Z Wrote inspect data to must-gather. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:20.446343775Z Inspecting resource clusterrole.rbac.authorization.k8s.io/istio-reader-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:20.471330145Z Wrote inspect data to must-gather. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:20.494532903Z tar: Removing leading `/' from member names [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:20.499627439Z INFO: HAProxy config collection complete. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:20.616294832Z Wrote inspect data to must-gather. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:20.644602286Z error: the server doesn't have a resource type "machineconfigs" [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:20.891544676Z INFO: Getting rules from prometheus-k8s-0 [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:20.940625450Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:21.003543383Z error: the server doesn't have a resource type "machineconfigpools" [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:21.098389076Z INFO: 'previous-logs' folder not found on ip-10-0-140-52.ec2.internal, skipping... [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:21.098428066Z INFO: Waiting for Machine Config Daemon termination log collection to complete ... [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:21.098456102Z INFO: Machine Config Daemon termination log collection complete. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:21.103952971Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:21.106290789Z Wrote inspect data to must-gather. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:21.108273878Z Inspecting resource clusterrole.rbac.authorization.k8s.io/istiod-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:21.412786869Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:21.506313970Z Wrote inspect data to must-gather. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:21.553186490Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:21.556586972Z Inspecting resource clusterrole.rbac.authorization.k8s.io/istiod-gateway-controller-openshift-gateway-openshift-ingress [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:21.617394696Z INFO: Getting status/config from prometheus-k8s-0 [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:21.735520571Z Wrote inspect data to must-gather. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:21.833985007Z error: the server doesn't have a resource type "kubeletconfigs" [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:21.844113190Z Gathering data for ns/openshift-monitoring... [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:21.927990679Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:22.015986715Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:22.019226532Z Inspecting resource clusterrolebinding.rbac.authorization.k8s.io/istio-reader-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:22.209674050Z Wrote inspect data to must-gather. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:22.259911280Z Wrote inspect data to must-gather. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:22.319654639Z INFO: Getting status/flags from prometheus-k8s-0 [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:22.404465681Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:22.405445209Z Inspecting resource clusterrolebinding.rbac.authorization.k8s.io/istiod-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:22.640019341Z INFO: Getting status/runtimeinfo from prometheus-k8s-0 [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:22.675286886Z INFO: INTERCONNECT MODE [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:22.677134280Z INFO: Gathering ovn-kubernetes DBs [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:22.681325790Z Wrote inspect data to must-gather. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:22.777049499Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:22.783936963Z Inspecting resource clusterrolebinding.rbac.authorization.k8s.io/istiod-gateway-controller-openshift-gateway-openshift-ingress [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:22.919326847Z INFO: Gathering OVN_Northbound from ovnkube-node-c954x... [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:22.919664827Z INFO: Gathering OVN_Northbound from ovnkube-node-kxrll... [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:22.919971394Z INFO: Gathering OVN_Northbound from ovnkube-node-qbd6q... [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:22.920471366Z INFO: Gathering OVN_Southbound from ovnkube-node-c954x... [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:22.920774818Z INFO: Gathering OVN_Southbound from ovnkube-node-kxrll... [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:22.921056821Z INFO: Gathering OVN_Southbound from ovnkube-node-qbd6q... [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:22.967060800Z INFO: Getting targets?state=active from prometheus-k8s-0 [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:23.159574317Z Wrote inspect data to must-gather. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:23.237244415Z tar: Removing leading `/' from member names [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:23.264231983Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:23.358765420Z tar: Removing leading `/' from member names [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:23.378511398Z tar: Removing leading `/' from member names [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:23.438915990Z tar: Removing leading `/' from member names [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:23.472727597Z tar: Removing leading `/' from member names [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:23.481742557Z tar: Removing leading `/' from member names [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:23.544806019Z INFO: Getting status/tsdb from prometheus-k8s-0 [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:23.656745167Z Wrote inspect data to must-gather. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:24.028817064Z Wrote inspect data to must-gather. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:24.102864985Z INFO: Getting status from alertmanager-main-0 [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:24.193112079Z INFO: Waiting for network log collection to complete ... [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:24.193154984Z INFO: Waiting for ovnk database copies to complete ... [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:24.193531615Z INFO: Copying ovnk databases complete. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:24.195653747Z 6.8M must-gather/network_logs/ovnk_database_store [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:24.198380270Z ovnk_database_store/ [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:24.198380270Z ovnk_database_store/ovnkube-node-c954x_sbdb [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:24.243758347Z ovnk_database_store/ovnkube-node-kxrll_sbdb [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:24.290249735Z ovnk_database_store/ovnkube-node-qbd6q_sbdb [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:24.351872064Z ovnk_database_store/ovnkube-node-kxrll_nbdb [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:24.366850455Z ovnk_database_store/ovnkube-node-c954x_nbdb [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:24.383254988Z ovnk_database_store/ovnkube-node-qbd6q_nbdb [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:24.406966121Z INFO: Network log collection complete. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:24.514620383Z Wrote inspect data to must-gather. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:24.574129249Z [disk usage checker] Volume usage percentage: current = 21 ; allowed = 70 [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:24.599809487Z Gathering data for ns/openshift-network-console... [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:24.894649210Z Wrote inspect data to must-gather. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:24.960994460Z Wrote inspect data to must-gather. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:25.230465901Z Gathering data for ns/openshift-console-operator... [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:25.237722887Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:25.240122440Z INFO: Fallback to identify the container image from release info [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:25.675080809Z Gathering data for ns/openshift-console... [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:25.701782722Z Inspecting resource crd/authorizationpolicies.security.istio.io [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:25.918496611Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:25.919909387Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811 [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:26.057086487Z daemonset.apps/perf-node-gather-daemonset created [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:26.155595309Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:26.159801917Z Inspecting resource crd/destinationrules.networking.istio.io [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:26.228820204Z Waiting for performance profile collector pods to become ready: 1 [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:26.359687155Z Gathering data for ns/openshift-cluster-storage-operator... [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:26.455316280Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:26.715157595Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:26.719012610Z Inspecting resource crd/envoyfilters.networking.istio.io [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:26.938715124Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:27.004270994Z Gathering data for ns/openshift-dns-operator... [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:27.181742036Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:27.184564845Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:27.392850338Z Waiting for performance profile collector pods to become ready: 2 [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:27.410750800Z Gathering data for ns/openshift-dns... [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:27.413010748Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:27.609115718Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:27.613220758Z Inspecting resource crd/gateways.gateway.networking.k8s.io [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:27.839429624Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:28.052765596Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:28.055941656Z Inspecting resource crd/gateways.networking.istio.io [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:28.214053068Z Gathering data for ns/openshift-image-registry... [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:28.267457643Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:28.510626887Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:28.513428923Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:28.562404526Z Waiting for performance profile collector pods to become ready: 3 [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:28.720466501Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:28.855912388Z Gathering data for ns/openshift-ingress-operator... [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:28.969926791Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:28.972385615Z Inspecting resource crd/httproutes.gateway.networking.k8s.io [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:29.213835107Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:29.294126041Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:29.481054402Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:29.484787574Z Inspecting resource crd/inferencepools.inference.networking.k8s.io [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:29.581677309Z [disk usage checker] Volume usage percentage: current = 21 ; allowed = 70 [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:29.721119251Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:29.747168464Z Waiting for performance profile collector pods to become ready: 4 [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:29.959534350Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:29.962211101Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:30.154851568Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:30.392254296Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:30.394885520Z Inspecting resource crd/peerauthentications.security.istio.io [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:30.457112974Z Gathering data for ns/openshift-ingress-canary... [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:30.596776008Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:30.846632207Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:30.849222568Z Inspecting resource crd/proxyconfigs.networking.istio.io [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:30.912732674Z Waiting for performance profile collector pods to become ready: 5 [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:30.961050753Z Gathering data for ns/openshift-insights... [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:31.053649859Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:31.269143169Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:31.272259399Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:31.457332788Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:31.690179001Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:31.692958828Z Inspecting resource crd/requestauthentications.security.istio.io [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:31.876584184Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:32.051194687Z Waiting for performance profile collector pods to become ready: 6 [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:32.095679371Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:32.099324810Z Inspecting resource crd/serviceentries.networking.istio.io [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:32.274959222Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:32.491181516Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:32.493503221Z Inspecting resource crd/sidecars.networking.istio.io [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:32.684852129Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:32.896203460Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:32.900041577Z Inspecting resource crd/telemetries.telemetry.istio.io [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:33.099614198Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:33.193903801Z Waiting for performance profile collector pods to become ready: 7 [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:33.330421107Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:33.332906523Z Inspecting resource crd/virtualservices.networking.istio.io [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:33.534644007Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:33.743772547Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:33.746474434Z Inspecting resource crd/wasmplugins.extensions.istio.io [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:33.924787829Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:34.046528839Z Gathering data for ns/openshift-lws-operator... [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:34.134057296Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:34.137555187Z Inspecting resource crd/workloadentries.networking.istio.io [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:34.347153211Z Daemonset perf-node-gather-daemonset ready 3 out of 3 [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:34.357795656Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:34.589213892Z [disk usage checker] Volume usage percentage: current = 21 ; allowed = 70 [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:34.625415699Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:34.628802617Z Inspecting resource crd/workloadgroups.networking.istio.io [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:34.680337167Z Collecting performance related data for node ip-10-0-140-52.ec2.internal [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:34.686727416Z Collecting performance related data for node ip-10-0-135-20.ec2.internal [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:34.695882258Z Collecting performance related data for node ip-10-0-135-188.ec2.internal [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:34.762097021Z Gathering data for ns/kserve... [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:34.872644322Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:35.111176957Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:35.315395772Z Inspecting resource mutatingwebhookconfiguration.admissionregistration.k8s.io/istio-sidecar-injector-openshift-gateway-openshift-ingress [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:35.587267890Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:36.445322553Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:36.699821698Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:36.852947920Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:37.010331114Z Inspecting resource validatingwebhookconfiguration.admissionregistration.k8s.io/istio-validator-openshift-gateway-openshift-ingress [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:37.236316789Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:37.609738712Z Gathering data for ns/openshift-config... [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:38.078776734Z Gathering data for ns/openshift-config-managed... [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:38.107310815Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:38.334736387Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:38.502378673Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:38.721144954Z Gathering data for ns/openshift-kube-apiserver-operator... [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:39.059942996Z Done executing Istio gather script [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:39.099158535Z Gathering data for ns/openshift-kube-apiserver... [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:39.440610973Z Gathering data for ns/openshift-kube-controller-manager... [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:39.594326822Z [disk usage checker] Volume usage percentage: current = 21 ; allowed = 70 [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:39.796413655Z Gathering data for ns/openshift-kube-controller-manager-operator... [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:40.154858282Z Gathering data for ns/openshift-kube-scheduler... [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:40.505656083Z Gathering data for ns/openshift-kube-scheduler-operator... [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:40.875428164Z Gathering data for ns/openshift-kube-storage-version-migrator... [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:41.257937565Z Gathering data for ns/openshift-kube-storage-version-migrator-operator... [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:41.636046570Z Gathering data for ns/openshift-user-workload-monitoring... [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:42.314153196Z Gathering data for ns/openshift-multus... [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:43.968522415Z Gathering data for ns/openshift-ovn-kubernetes... [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:44.245353376Z Collecting kubelet logs for node ip-10-0-140-52.ec2.internal [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:44.247090107Z Collecting kubelet logs for node ip-10-0-135-20.ec2.internal [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:44.248806673Z Collecting kubelet logs for node ip-10-0-135-188.ec2.internal [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:44.599165606Z [disk usage checker] Volume usage percentage: current = 21 ; allowed = 70 [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:44.664698516Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-ct488 namespace [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:44.670860830Z INFO: Node performance data collection complete. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:45.406347038Z Gathering data for ns/openshift-host-network... [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:45.744810789Z Gathering data for ns/openshift-network-diagnostics... [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:46.321362423Z Gathering data for ns/openshift-network-node-identity... [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:46.798157626Z Gathering data for ns/openshift-network-operator... [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:47.209138319Z Gathering data for ns/openshift-cloud-network-config-controller... [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:47.502339884Z Gathering data for ns/openshift-cluster-node-tuning-operator... [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:48.016437954Z Gathering data for ns/openshift-apiserver-operator... [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:48.354688543Z Gathering data for ns/openshift-apiserver... [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:48.817395818Z Gathering data for ns/openshift-controller-manager-operator... [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:49.130335840Z Gathering data for ns/openshift-controller-manager... [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:49.470333259Z Gathering data for ns/openshift-cluster-samples-operator... [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:49.607724020Z [disk usage checker] Volume usage percentage: current = 21 ; allowed = 70 [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:50.179714144Z Gathering data for ns/openshift-operator-lifecycle-manager... [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:50.527405978Z Gathering data for ns/openshift-service-ca-operator... [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:50.869240049Z Gathering data for ns/openshift-service-ca... [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:51.273264002Z Gathering data for ns/openshift-cluster-csi-drivers... [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:52.254468138Z Wrote inspect data to must-gather. [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:52.254515892Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:52.254515892Z skipping gathering secrets/support due to error: secrets "support" not found [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:52.399441543Z error: the server doesn't have a resource type "clusters" [must-gather] [must-gather-wlhgr] POD 2026-07-28T18:56:52.682641166Z Caches written to disk [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:53.692316506Z waiting for gather to complete [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:53.695878348Z downloading gather output [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:53.962164262Z receiving incremental file list [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:53.986864753Z ./ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:53.986996026Z aggregated-discovery-api.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:53.987347293Z aggregated-discovery-apis.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:53.988251952Z event-filter.html [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:53.991594969Z timestamp [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:53.991715502Z version [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.001609252Z cluster-scoped-resources/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.001628853Z cluster-scoped-resources/admissionregistration.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.001634633Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.001737605Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/default-network-annotation.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.001873858Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/servicecidrs.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.002006591Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/user-defined-networks-namespace-label.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.002069042Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.002141993Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/default-network-annotation-binding.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.002263516Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/servicecidrs-binding.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.002372028Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/user-defined-networks-namespace-label-binding.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.002526251Z cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.002573492Z cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/multus.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.002722495Z cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/network-node-identity.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.002815147Z cluster-scoped-resources/apiextensions.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.003297997Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.003342688Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/accounts.nim.opendatahub.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.003541282Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/adminnetworkpolicies.policy.networking.k8s.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.003803047Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/adminpolicybasedexternalroutes.k8s.ovn.org.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.004014601Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertingrules.monitoring.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.004224176Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertmanagerconfigs.monitoring.coreos.com.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.008033323Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertmanagers.monitoring.coreos.com.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.009601665Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertrelabelconfigs.monitoring.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.009747148Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apikeyapprovals.devportal.kuadrant.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.009932721Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apikeyrequests.devportal.kuadrant.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.010053984Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apikeys.devportal.kuadrant.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.010221877Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apiproducts.devportal.kuadrant.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.010726477Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apirequestcounts.apiserver.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.010871531Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apiservers.config.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.011035414Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/appliedmanifestworks.work.open-cluster-management.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.011145186Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authconfigs.authorino.kuadrant.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.011923982Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authentications.config.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.012187437Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authorinos.operator.authorino.kuadrant.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.01231801Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authorizationpolicies.security.istio.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.012491013Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authpolicies.kuadrant.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.013773069Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/baselineadminnetworkpolicies.policy.networking.k8s.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.014006824Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/builds.config.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.014200318Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/bundles.trust.cert-manager.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.014355521Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/catalogsources.operators.coreos.com.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.014664537Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/certificaterequests.cert-manager.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.014820351Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/certificates.cert-manager.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.015015004Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/certmanagers.operator.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.015321401Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/challenges.acme.cert-manager.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.015961564Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/cloudcredentials.operator.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.016081336Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/cloudeventsources.eventing.keda.sh.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.016220409Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/cloudprivateipconfigs.cloud.network.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.016364722Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterclaims.cluster.open-cluster-management.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.016468414Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clustercloudeventsources.eventing.keda.sh.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.016579346Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clustercsidrivers.operator.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.01676813Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterimagepolicies.config.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.016958384Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterissuers.cert-manager.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.01774075Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusteroperators.config.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.017868513Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterresourcequotas.quota.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.018047416Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterserviceversions.operators.coreos.com.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.020036357Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterstoragecontainers.serving.kserve.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.02021558Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clustertriggerauthentications.keda.sh.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.020366293Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusteruserdefinednetworks.k8s.ovn.org.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.020600848Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterversions.config.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.020871634Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/configs.imageregistry.operator.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.021312672Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/configs.operator.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.021429825Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/configs.samples.operator.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.021555117Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleclidownloads.console.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.021657919Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleexternalloglinks.console.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.021777982Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolelinks.console.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.021894584Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolenotifications.console.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.022021377Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleplugins.console.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.022199641Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolequickstarts.console.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.022336003Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoles.config.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.022449106Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoles.operator.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.022737241Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolesamples.console.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.022901855Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleyamlsamples.console.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.023008307Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/controllerconfigs.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.023572358Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/credentialsrequests.cloudcredential.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.023700731Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/csisnapshotcontrollers.operator.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.023813553Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/datascienceclusters.datasciencecluster.opendatahub.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.024025358Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/destinationrules.networking.istio.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.024794683Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnses.config.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.024912945Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnses.operator.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.025098349Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnshealthcheckprobes.kuadrant.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.025235252Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnspolicies.kuadrant.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.025403146Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnsrecords.ingress.operator.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.025552258Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnsrecords.kuadrant.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.025717502Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dscinitializations.dscinitialization.opendatahub.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.025854045Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressfirewalls.k8s.ovn.org.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.025977317Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressips.k8s.ovn.org.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.026094789Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressqoses.k8s.ovn.org.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.026260233Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressrouters.network.operator.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.026391126Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressservices.k8s.ovn.org.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.026516358Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/envoyfilters.networking.istio.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.026685412Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/etcds.operator.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.026825124Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/featuregates.config.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.026967097Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gatewayclasses.gateway.networking.k8s.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.02711384Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.gateway.networking.k8s.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.02761042Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.networking.istio.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.027777923Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/grpcroutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.028222183Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/helmchartrepositories.helm.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.028327885Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/httproutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.029117221Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/httpscaledobjects.http.keda.sh.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.029296194Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagecontentpolicies.config.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.029410687Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagecontentsourcepolicies.operator.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.029519509Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagedigestmirrorsets.config.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.029644652Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagepolicies.config.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.029840825Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagepruners.imageregistry.operator.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.03008211Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/images.config.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.030232403Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagetagmirrorsets.config.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.030347086Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencegraphs.serving.kserve.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.030505099Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencemodelrewrites.llm-d.ai.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.030637372Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferenceobjectives.llm-d.ai.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.030743494Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.k8s.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.030889097Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.0310355Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferenceservices.serving.kserve.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.034954059Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/infrastructures.config.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.035392678Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ingresscontrollers.operator.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.03597295Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ingresses.config.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.036151693Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/insightsoperators.operator.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.036300967Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/installplans.operators.coreos.com.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.036433229Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/interceptorroutes.http.keda.sh.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.036568822Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ipamclaims.k8s.cni.cncf.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.036681894Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ippools.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.036779406Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/issuers.cert-manager.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.037596753Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/istiocsrs.operator.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.037890539Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kedacontrollers.keda.sh.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.041792368Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kuadrants.kuadrant.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.04191639Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubeapiservers.operator.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.042061333Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubecontrollermanagers.operator.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.042295158Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubeschedulers.operator.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.042429261Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubestorageversionmigrators.operator.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.042541943Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/leaderworkersetoperators.operator.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.042668326Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/leaderworkersets.leaderworkerset.x-k8s.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.045269738Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/limitadors.limitador.kuadrant.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.048199398Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/llminferenceserviceconfigs.serving.kserve.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.053265331Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/llminferenceservices.serving.kserve.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.058437326Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineconfignodes.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.058605239Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineconfigurations.operator.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.058913686Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineosbuilds.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.059076369Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineosconfigs.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.059247902Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/network-attachment-definitions.k8s.cni.cncf.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.059350874Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/networks.config.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.059509587Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/networks.operator.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.059848254Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/nodes.config.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.059961127Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/nodeslicepools.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.060063509Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/oauths.config.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.060280223Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/oidcpolicies.extensions.kuadrant.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.060413626Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/olmconfigs.operators.coreos.com.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.060530668Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/openshiftapiservers.operator.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.060651931Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/openshiftcontrollermanagers.operator.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.060759523Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorconditions.operators.coreos.com.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.060894706Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorgroups.operators.coreos.com.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.061049789Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorhubs.config.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.061187562Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorpkis.network.operator.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.061296644Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operators.operators.coreos.com.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.061417576Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/orders.acme.cert-manager.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.061578589Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/overlappingrangeipreservations.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.061687562Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/peerauthentications.security.istio.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.061869785Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/pinnedimagesets.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.062019329Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/planpolicies.extensions.kuadrant.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.062145561Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/podmonitors.monitoring.coreos.com.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.062455517Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/podnetworkconnectivitychecks.controlplane.operator.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.06259534Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/probes.monitoring.coreos.com.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.062848715Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/profiles.tuned.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.062960988Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/projecthelmchartrepositories.helm.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.06307816Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/projects.config.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.063202653Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/prometheuses.monitoring.coreos.com.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.065234814Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/prometheusrules.monitoring.coreos.com.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.065362676Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/proxies.config.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.065472399Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/proxyconfigs.networking.istio.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.065589801Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/rangeallocations.security.internal.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.065694333Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ratelimitpolicies.kuadrant.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.065832106Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/referencegrants.gateway.networking.k8s.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.065948998Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/requestauthentications.security.istio.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.066097881Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/rolebindingrestrictions.authorization.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.066246294Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/scaledjobs.keda.sh.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.067802636Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/scaledobjects.keda.sh.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.067962519Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/schedulers.config.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.068102282Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/securitycontextconstraints.security.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.068274485Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/servicecas.operator.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.068389068Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/serviceentries.networking.istio.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.068564661Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/servicemonitors.monitoring.coreos.com.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.068809246Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/servingruntimes.serving.kserve.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.069423019Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/sidecars.networking.istio.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.069691564Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/storages.operator.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.069818247Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/storagestates.migration.k8s.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.069927379Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/storageversionmigrations.migration.k8s.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.070036691Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/subscriptions.operators.coreos.com.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.070684154Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/telemetries.telemetry.istio.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.070896629Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/telemetrypolicies.extensions.kuadrant.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.071003941Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/thanosrulers.monitoring.coreos.com.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.072584703Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/tlspolicies.kuadrant.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.072764736Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/tokenratelimitpolicies.kuadrant.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.073021392Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/trainedmodels.serving.kserve.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.073145014Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/triggerauthentications.keda.sh.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.073377839Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/trustmanagers.operator.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.073689745Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/tuneds.tuned.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.073848078Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/userdefinednetworks.k8s.ovn.org.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.074071153Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/virtualservices.networking.istio.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.074628904Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumepopulators.populator.storage.k8s.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.074756097Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumesnapshotclasses.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.074968351Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumesnapshotcontents.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.075208336Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumesnapshots.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.0754081Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/wasmplugins.extensions.istio.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.075574673Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadentries.networking.istio.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.075691256Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadgroups.networking.istio.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.07591119Z cluster-scoped-resources/apiregistration.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.076253637Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.076292288Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1..yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.076418331Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.acme.cert-manager.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.076513513Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.admissionregistration.k8s.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.076619755Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apiextensions.k8s.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.076709107Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apiserver.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.07688914Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apps.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.077032763Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apps.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.077145905Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.authentication.k8s.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.077264488Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.authorization.k8s.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.077345809Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.authorization.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.077447191Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.autoscaling.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.077570174Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.batch.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.077678226Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.build.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.077761838Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.cert-manager.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.07784717Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.certificates.k8s.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.077933021Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.cloud.network.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.078043254Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.cloudcredential.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.078185197Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.config.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.078300459Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.console.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.078414621Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.coordination.k8s.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.078533544Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.datasciencecluster.opendatahub.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.078648586Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.discovery.k8s.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.078769718Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.dscinitialization.opendatahub.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.078879861Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.events.k8s.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.078987073Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.flowcontrol.apiserver.k8s.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.079096015Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.gateway.networking.k8s.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.079222487Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.image.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.0793252Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.imageregistry.operator.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.079412402Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.inference.networking.k8s.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.079505843Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.ingress.operator.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.079582975Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.k8s.cni.cncf.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.079676507Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.k8s.ovn.org.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.079772519Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.kuadrant.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.07985492Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.leaderworkerset.x-k8s.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.079940472Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.080031644Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.monitoring.coreos.com.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.080142476Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.monitoring.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.080235318Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.network.operator.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.08035609Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.networking.istio.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.080496733Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.networking.k8s.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.080606046Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.nim.opendatahub.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.080729928Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.node.k8s.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.08083964Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.oauth.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.080930702Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.operator.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.081021194Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.operators.coreos.com.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.081109966Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.packages.operators.coreos.com.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.081230498Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.policy.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.081352531Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.project.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.081449043Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.quota.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.081538705Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.rbac.authorization.k8s.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.081621926Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.resource.k8s.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.081722948Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.route.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.081862451Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.samples.operator.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.081946423Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.scheduling.k8s.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.082053065Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.security.internal.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.082175767Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.security.istio.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.08227002Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.security.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.082365181Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.082473344Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.storage.k8s.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.082559285Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.telemetry.istio.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.082687548Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.template.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.08278236Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.tuned.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.082870481Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.user.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.082974464Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.work.open-cluster-management.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.083082916Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.cluster.open-cluster-management.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.083206668Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.controlplane.operator.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.08329705Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.devportal.kuadrant.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.083407252Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.eventing.keda.sh.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.083483004Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.extensions.istio.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.083590676Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.extensions.kuadrant.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.083678568Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.http.keda.sh.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.0837727Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.k8s.cni.cncf.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.083860502Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.keda.sh.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.083963964Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.kuadrant.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.084053766Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.limitador.kuadrant.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.084223379Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.migration.k8s.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.084329701Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.monitoring.coreos.com.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.084409823Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.operator.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.084505215Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.operators.coreos.com.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.084605947Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.policy.networking.k8s.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.084697279Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.serving.kserve.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.08479196Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.telemetry.istio.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.084879352Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.trust.cert-manager.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.084964494Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.085057336Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.085173048Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.llm-d.ai.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.085282281Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.operators.coreos.com.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.085374912Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.serving.kserve.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.085465224Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha3.networking.istio.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.085553046Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.admissionregistration.k8s.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.085641388Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.external.metrics.k8s.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.08575714Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.gateway.networking.k8s.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.085849072Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.helm.openshift.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.085942944Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.http.keda.sh.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.086039046Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.kuadrant.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.086150768Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.metrics.k8s.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.086265261Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.monitoring.coreos.com.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.086371573Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.networking.istio.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.086488495Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.operator.authorino.kuadrant.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.086606928Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.populator.storage.k8s.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.08672515Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.security.istio.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.086834732Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.serving.kserve.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.086945904Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.storage.k8s.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.087056766Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta3.authorino.kuadrant.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.087181829Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v2.autoscaling.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.087299442Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v2.operators.coreos.com.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.087370023Z cluster-scoped-resources/certificates.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.087384193Z cluster-scoped-resources/certificates.k8s.io/certificatesigningrequests/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.087444894Z cluster-scoped-resources/certificates.k8s.io/certificatesigningrequests/system:openshift:openshift-monitoring-l4pfl.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.087603028Z cluster-scoped-resources/certificates.k8s.io/certificatesigningrequests/system:openshift:openshift-monitoring-wkwr5.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.08773377Z cluster-scoped-resources/certificates.k8s.io/certificatesigningrequests/system:openshift:openshift-monitoring-xs2m5.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.087989945Z cluster-scoped-resources/config.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.088075567Z cluster-scoped-resources/config.openshift.io/apiservers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.088323672Z cluster-scoped-resources/config.openshift.io/authentications.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.088498036Z cluster-scoped-resources/config.openshift.io/builds.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.088662839Z cluster-scoped-resources/config.openshift.io/clusterimagepolicies.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.088757871Z cluster-scoped-resources/config.openshift.io/clusteroperators.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.089139469Z cluster-scoped-resources/config.openshift.io/clusterversions.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.089277442Z cluster-scoped-resources/config.openshift.io/consoles.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.089381944Z cluster-scoped-resources/config.openshift.io/dnses.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.089486326Z cluster-scoped-resources/config.openshift.io/featuregates.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.089618089Z cluster-scoped-resources/config.openshift.io/imagecontentpolicies.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.089742901Z cluster-scoped-resources/config.openshift.io/imagedigestmirrorsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.089836373Z cluster-scoped-resources/config.openshift.io/imagepolicies.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.089931015Z cluster-scoped-resources/config.openshift.io/images.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.090043947Z cluster-scoped-resources/config.openshift.io/imagetagmirrorsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.090143229Z cluster-scoped-resources/config.openshift.io/infrastructures.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.090276222Z cluster-scoped-resources/config.openshift.io/ingresses.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.090377654Z cluster-scoped-resources/config.openshift.io/networks.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.090486236Z cluster-scoped-resources/config.openshift.io/nodes.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.090593268Z cluster-scoped-resources/config.openshift.io/oauths.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.090703161Z cluster-scoped-resources/config.openshift.io/operatorhubs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.090794742Z cluster-scoped-resources/config.openshift.io/projects.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.090898344Z cluster-scoped-resources/config.openshift.io/proxies.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.091008727Z cluster-scoped-resources/config.openshift.io/schedulers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.091142309Z cluster-scoped-resources/config.openshift.io/clusteroperators/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.091246662Z cluster-scoped-resources/config.openshift.io/clusteroperators/console.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.091366034Z cluster-scoped-resources/config.openshift.io/clusteroperators/csi-snapshot-controller.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.091482436Z cluster-scoped-resources/config.openshift.io/clusteroperators/dns.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.091585169Z cluster-scoped-resources/config.openshift.io/clusteroperators/image-registry.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.09168312Z cluster-scoped-resources/config.openshift.io/clusteroperators/ingress.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.091791823Z cluster-scoped-resources/config.openshift.io/clusteroperators/insights.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.091942546Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-apiserver.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.09215269Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-controller-manager.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.092266432Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-scheduler.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.092375274Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-storage-version-migrator.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.092476676Z cluster-scoped-resources/config.openshift.io/clusteroperators/monitoring.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.092621199Z cluster-scoped-resources/config.openshift.io/clusteroperators/network.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.092777303Z cluster-scoped-resources/config.openshift.io/clusteroperators/node-tuning.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.092880055Z cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-apiserver.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.093020807Z cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-controller-manager.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.09313482Z cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-samples.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.093295033Z cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager-catalog.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.093415826Z cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager-packageserver.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.093523008Z cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.09363495Z cluster-scoped-resources/config.openshift.io/clusteroperators/service-ca.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.093763953Z cluster-scoped-resources/config.openshift.io/clusteroperators/storage.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.093845784Z cluster-scoped-resources/config.openshift.io/clusterversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.0941013Z cluster-scoped-resources/config.openshift.io/clusterversions/version.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.094223572Z cluster-scoped-resources/config.openshift.io/consoles/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.094304354Z cluster-scoped-resources/config.openshift.io/consoles/cluster.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.094410436Z cluster-scoped-resources/config.openshift.io/featuregates/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.094468197Z cluster-scoped-resources/config.openshift.io/featuregates/cluster.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.094554039Z cluster-scoped-resources/config.openshift.io/infrastructures/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.09462282Z cluster-scoped-resources/config.openshift.io/infrastructures/cluster.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.094691772Z cluster-scoped-resources/config.openshift.io/oauths/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.094754163Z cluster-scoped-resources/config.openshift.io/oauths/cluster.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.094805984Z cluster-scoped-resources/config.openshift.io/proxies/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.094866375Z cluster-scoped-resources/config.openshift.io/proxies/cluster.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.094921736Z cluster-scoped-resources/console.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.094933466Z cluster-scoped-resources/console.openshift.io/consoleplugins/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.094965317Z cluster-scoped-resources/console.openshift.io/consoleplugins/kuadrant-console-plugin.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.095073389Z cluster-scoped-resources/console.openshift.io/consoleplugins/monitoring-plugin.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.095196902Z cluster-scoped-resources/console.openshift.io/consoleplugins/networking-console-plugin.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.095254643Z cluster-scoped-resources/core/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.095261813Z cluster-scoped-resources/core/nodes/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.095316914Z cluster-scoped-resources/core/nodes/ip-10-0-135-188.ec2.internal.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.095507558Z cluster-scoped-resources/core/nodes/ip-10-0-135-20.ec2.internal.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.095656301Z cluster-scoped-resources/core/nodes/ip-10-0-140-52.ec2.internal.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.095762913Z cluster-scoped-resources/core/persistentvolumes/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.095813334Z cluster-scoped-resources/core/persistentvolumes/pvc-f1df9520-ed9a-482b-9b2b-0f8681809f16.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.095901366Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.095965447Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.096050909Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/catch-all.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.096201792Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/exempt.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.096302614Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/global-default.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.096392626Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/kube-controller-manager.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.096505348Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/kube-scheduler.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.09658951Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/kube-system-service-accounts.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.096758583Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-apiserver-operator.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.096869516Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-apiserver-sar.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.096978308Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-apiserver.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.09707574Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-authentication-operator.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.097211083Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-controller-manager.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.097326465Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-kube-apiserver-operator.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.097433097Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-monitoring-metrics.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.097532889Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-oauth-apiserver-sar.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.097639121Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-oauth-apiserver.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.097736713Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-oauth-server.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.097842145Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/probes.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.097950838Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/service-accounts.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.09805327Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/system-leader-election.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.098216963Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/system-node-high.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.098311265Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/system-nodes.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.098421657Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.098471158Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/catch-all.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.098637121Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/exempt.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.098713633Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/global-default.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.098817285Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/leader-election.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.098917487Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/node-high.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.09902182Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/openshift-control-plane-operators.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.099246124Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/system.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.099406047Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/workload-high.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.099511019Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/workload-low.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.099571751Z cluster-scoped-resources/gateway.networking.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.099580891Z cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.099635032Z cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/openshift-default.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.099720403Z cluster-scoped-resources/imageregistry.operator.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.099733144Z cluster-scoped-resources/imageregistry.operator.openshift.io/configs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.099793745Z cluster-scoped-resources/imageregistry.operator.openshift.io/configs/cluster.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.099896477Z cluster-scoped-resources/imageregistry.operator.openshift.io/imagepruners/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.099982369Z cluster-scoped-resources/imageregistry.operator.openshift.io/imagepruners/cluster.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.10005528Z cluster-scoped-resources/migration.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.100067101Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.100108731Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/console-plugin-storage-version-migration.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.100254414Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/machineconfiguration-controllerconfig-storage-version-migration.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.100433788Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/machineconfiguration-machineconfigpool-storage-version-migration.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.100493229Z cluster-scoped-resources/oauth.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.100501329Z cluster-scoped-resources/oauth.openshift.io/oauthclients/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.10054257Z cluster-scoped-resources/oauth.openshift.io/oauthclients/console.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.100646582Z cluster-scoped-resources/operator.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.100654962Z cluster-scoped-resources/operator.openshift.io/clustercsidrivers/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.100698043Z cluster-scoped-resources/operator.openshift.io/clustercsidrivers/ebs.csi.aws.com.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.100802845Z cluster-scoped-resources/operator.openshift.io/consoles/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.100853086Z cluster-scoped-resources/operator.openshift.io/consoles/cluster.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.101393537Z cluster-scoped-resources/operator.openshift.io/csisnapshotcontrollers/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.101437518Z cluster-scoped-resources/operator.openshift.io/csisnapshotcontrollers/cluster.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.101555101Z cluster-scoped-resources/operator.openshift.io/dnses/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.101601022Z cluster-scoped-resources/operator.openshift.io/dnses/default.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.101662723Z cluster-scoped-resources/operator.openshift.io/insightsoperators/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.101701844Z cluster-scoped-resources/operator.openshift.io/insightsoperators/cluster.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.101813816Z cluster-scoped-resources/operator.openshift.io/kubeapiservers/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.101854427Z cluster-scoped-resources/operator.openshift.io/kubeapiservers/cluster.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.101911258Z cluster-scoped-resources/operator.openshift.io/kubecontrollermanagers/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.102036401Z cluster-scoped-resources/operator.openshift.io/kubecontrollermanagers/cluster.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.102081711Z cluster-scoped-resources/operator.openshift.io/kubeschedulers/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.102106562Z cluster-scoped-resources/operator.openshift.io/kubeschedulers/cluster.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.102198384Z cluster-scoped-resources/operator.openshift.io/kubestorageversionmigrators/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.102238765Z cluster-scoped-resources/operator.openshift.io/kubestorageversionmigrators/cluster.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.102305186Z cluster-scoped-resources/operator.openshift.io/networks/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.102348287Z cluster-scoped-resources/operator.openshift.io/networks/cluster.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.102418138Z cluster-scoped-resources/operator.openshift.io/openshiftapiservers/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.102464769Z cluster-scoped-resources/operator.openshift.io/openshiftapiservers/cluster.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.102524371Z cluster-scoped-resources/operator.openshift.io/openshiftcontrollermanagers/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.102571732Z cluster-scoped-resources/operator.openshift.io/openshiftcontrollermanagers/cluster.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.102654393Z cluster-scoped-resources/operator.openshift.io/servicecas/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.102683824Z cluster-scoped-resources/operator.openshift.io/servicecas/cluster.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.102756715Z cluster-scoped-resources/operator.openshift.io/storages/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.102807556Z cluster-scoped-resources/operator.openshift.io/storages/cluster.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.102899648Z cluster-scoped-resources/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.102908728Z cluster-scoped-resources/operators.coreos.com/olmconfigs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.102953869Z cluster-scoped-resources/operators.coreos.com/olmconfigs/cluster.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.103041651Z cluster-scoped-resources/operators.coreos.com/operators/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.103058951Z cluster-scoped-resources/operators.coreos.com/operators/authorino-operator.kuadrant-system.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.103206054Z cluster-scoped-resources/operators.coreos.com/operators/dns-operator.kuadrant-system.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.103378178Z cluster-scoped-resources/operators.coreos.com/operators/leader-worker-set.openshift-lws-operator.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.103535811Z cluster-scoped-resources/operators.coreos.com/operators/limitador-operator.kuadrant-system.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.103643223Z cluster-scoped-resources/operators.coreos.com/operators/openshift-cert-manager-operator.cert-manager-operator.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.103760916Z cluster-scoped-resources/operators.coreos.com/operators/openshift-custom-metrics-autoscaler-operator.openshift-keda.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.103867018Z cluster-scoped-resources/operators.coreos.com/operators/rhcl-operator.kuadrant-system.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.1039565Z cluster-scoped-resources/rbac.authorization.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.104066072Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.104076542Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/aws-ebs-csi-driver-operator-clusterrolebinding.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.104172614Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/cloud-network-config-controller.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.104264606Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/cluster-node-tuning-operator.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.104430759Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/cluster-storage-operator-role.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.104561492Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/csi-snapshot-controller-operator-clusterrole.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.104680844Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/csi-snapshot-controller-runner-operator.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.104791046Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/metrics-daemon-sa-rolebinding.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.104848598Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-admission-controller-webhook.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.104926419Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-ancillary-tools.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.105013121Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-cluster-readers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.105108403Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-group.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.105228765Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-transient.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.105412109Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-whereabouts.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.105518731Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/network-diagnostics.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.105610563Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/network-node-identity.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.105626923Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-image-registry-pruner.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.105651314Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-iptables-alerter.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.105735376Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.105820997Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-ovn-kubernetes-node-identity-limited.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.105907389Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-ovn-kubernetes-node-kube-rbac-proxy.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.105993111Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/registry-registry-role.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.106136524Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.106187625Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/aws-ebs-csi-driver-operator-clusterrole.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.106288387Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/cloud-network-config-controller.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.106372138Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/cluster-node-tuning-operator.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.1064626Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/console-extensions-reader.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.106664244Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/console-operator.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.106774917Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/console.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.106883509Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/csi-snapshot-controller-operator-clusterrole.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.107010751Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/helm-chartrepos-viewer.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.107108844Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/metrics-daemon-role.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.107269917Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus-admission-controller-webhook.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.107367379Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus-ancillary-tools.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.107523472Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.107622484Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/net-attach-def-project.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.107709446Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/network-diagnostics.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.107814008Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/network-node-identity.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.1079039Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-csi-snapshot-controller-runner.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.107997671Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-iptables-alerter.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.108093664Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-cluster-reader.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.108209256Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.108317908Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-kube-rbac-proxy.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.10839888Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-node-limited.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.108489652Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-udn-editor.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.108580433Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-udn-viewer.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.108675145Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/project-helm-chartrepository-editor.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.108764987Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-admin.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.108853289Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-basic-user.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.108958221Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-storage-admin.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.109046823Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-view.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.109147035Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:registry.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.109264577Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/whereabouts-cni.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.109307548Z cluster-scoped-resources/samples.operator.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.109317528Z cluster-scoped-resources/samples.operator.openshift.io/configs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.109359069Z cluster-scoped-resources/samples.operator.openshift.io/configs/cluster.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.1094168Z cluster-scoped-resources/snapshot.storage.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.109424931Z cluster-scoped-resources/snapshot.storage.k8s.io/volumesnapshotclasses/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.109442931Z cluster-scoped-resources/snapshot.storage.k8s.io/volumesnapshotclasses/csi-aws-vsc.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.109529482Z cluster-scoped-resources/storage.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.109540913Z cluster-scoped-resources/storage.k8s.io/csidrivers/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.109579183Z cluster-scoped-resources/storage.k8s.io/csidrivers/ebs.csi.aws.com.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.109642365Z cluster-scoped-resources/storage.k8s.io/csinodes/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.109698586Z cluster-scoped-resources/storage.k8s.io/csinodes/ip-10-0-135-188.ec2.internal.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.109783058Z cluster-scoped-resources/storage.k8s.io/csinodes/ip-10-0-135-20.ec2.internal.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.10987402Z cluster-scoped-resources/storage.k8s.io/csinodes/ip-10-0-140-52.ec2.internal.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.109929241Z cluster-scoped-resources/storage.k8s.io/storageclasses/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.109965802Z cluster-scoped-resources/storage.k8s.io/storageclasses/gp2-csi.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.110075194Z cluster-scoped-resources/storage.k8s.io/storageclasses/gp3-csi.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.110452521Z cluster-scoped-resources/storage.k8s.io/volumeattachments/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.110506082Z cluster-scoped-resources/storage.k8s.io/volumeattachments/csi-5bcc2ce21c4b5c028913383f0b9bc17d78d411bd4026f7b723692f4df21d86c1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.110569754Z host_service_logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.110581934Z host_service_logs/masters/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.110645635Z host_service_logs/masters/NetworkManager_service.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.110717087Z host_service_logs/masters/crio_service.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.110786048Z host_service_logs/masters/kubelet_service.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.110848079Z host_service_logs/masters/machine-config-daemon-firstboot_service.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.110987042Z host_service_logs/masters/machine-config-daemon-host_service.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.111099214Z host_service_logs/masters/openvswitch_service.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.111191136Z host_service_logs/masters/ostree-finalize-staged_service.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.111287438Z host_service_logs/masters/ovs-configuration_service.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.11138358Z host_service_logs/masters/ovs-vswitchd_service.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.111468742Z host_service_logs/masters/ovsdb-server_service.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.111566484Z host_service_logs/masters/rpm-ostreed_service.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.111602035Z ingress_controllers/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.111611535Z ingress_controllers/default/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.111615555Z ingress_controllers/default/router-default-66dc97f78c-hsl5f/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.111677746Z ingress_controllers/default/router-default-66dc97f78c-hsl5f/haproxy.config [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.111803799Z insights-data/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.111835149Z istio/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.111892831Z istio/aggregated-discovery-api.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.112010303Z istio/aggregated-discovery-apis.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.112506673Z istio/event-filter.html [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.112646966Z istio/timestamp [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.112775759Z istio/cluster-scoped-resources/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.112782779Z istio/cluster-scoped-resources/admissionregistration.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.112791909Z istio/cluster-scoped-resources/admissionregistration.k8s.io/mutatingwebhookconfigurations/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.1128444Z istio/cluster-scoped-resources/admissionregistration.k8s.io/mutatingwebhookconfigurations/istio-sidecar-injector-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.112924022Z istio/cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.112959962Z istio/cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/istio-validator-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.113033124Z istio/cluster-scoped-resources/apiextensions.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.113064844Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.113135506Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authorizationpolicies.security.istio.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.113306729Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/destinationrules.networking.istio.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.114063974Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/envoyfilters.networking.istio.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.11435196Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gatewayclasses.gateway.networking.k8s.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.114517084Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.gateway.networking.k8s.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.114933912Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.networking.istio.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.115105636Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/grpcroutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.115517474Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/httproutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.116548105Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.k8s.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.116700558Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.116834971Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/peerauthentications.security.istio.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.116961953Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/proxyconfigs.networking.istio.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.117066016Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/referencegrants.gateway.networking.k8s.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.117220549Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/requestauthentications.security.istio.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.117379812Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/serviceentries.networking.istio.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.117630857Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/sidecars.networking.istio.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.117925083Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/telemetries.telemetry.istio.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.118107547Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/virtualservices.networking.istio.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.118534735Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/wasmplugins.extensions.istio.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.118674428Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadentries.networking.istio.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.118812231Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadgroups.networking.istio.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.118960454Z istio/cluster-scoped-resources/gateway.networking.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.118969764Z istio/cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.118994525Z istio/cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/openshift-default.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.119064316Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.119077526Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.119153388Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/istio-reader-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.11924616Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/istiod-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.119423253Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/istiod-gateway-controller-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.119490095Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.119527896Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/istio-reader-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.119640218Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/istiod-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.119802421Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/istiod-gateway-controller-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.119868132Z istio/namespaces/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.119875463Z istio/namespaces/e2e-test-llm-auth-enabled-requires-token-d4ed2c08/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.119882203Z istio/namespaces/e2e-test-llm-auth-enabled-requires-token-d4ed2c08/gateway.networking.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.119886303Z istio/namespaces/e2e-test-llm-auth-enabled-requires-token-d4ed2c08/gateway.networking.k8s.io/httproutes/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.119928864Z istio/namespaces/e2e-test-llm-auth-enabled-requires-token-d4ed2c08/gateway.networking.k8s.io/httproutes/auth-enabled-test-kserve-route.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.120067096Z istio/namespaces/e2e-test-llm-auth-enabled-requires-token-d4ed2c08/inference.networking.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.120078417Z istio/namespaces/e2e-test-llm-auth-enabled-requires-token-d4ed2c08/inference.networking.k8s.io/inferencepools/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.120085447Z istio/namespaces/e2e-test-llm-auth-enabled-requires-token-d4ed2c08/inference.networking.k8s.io/inferencepools/auth-enabled-test-inference-pool.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.120187249Z istio/namespaces/e2e-test-llm-auth-enabled-requires-token-d4ed2c08/inference.networking.x-k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.120203919Z istio/namespaces/e2e-test-llm-auth-enabled-requires-token-d4ed2c08/inference.networking.x-k8s.io/inferencepools/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.12023278Z istio/namespaces/e2e-test-llm-auth-enabled-requires-token-d4ed2c08/inference.networking.x-k8s.io/inferencepools/auth-enabled-test-inference-pool.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.120316432Z istio/namespaces/e2e-test-llm-auth-enabled-requires-token-d4ed2c08/networking.istio.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.120328932Z istio/namespaces/e2e-test-llm-auth-enabled-requires-token-d4ed2c08/networking.istio.io/destinationrules/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.120368283Z istio/namespaces/e2e-test-llm-auth-enabled-requires-token-d4ed2c08/networking.istio.io/destinationrules/auth-enabled-test-kserve-scheduler.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.120465944Z istio/namespaces/e2e-test-llm-auth-enabled-requires-token-d4ed2c08/networking.istio.io/destinationrules/auth-enabled-test-kserve-shadow-svc.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.120545846Z istio/namespaces/e2e-test-llm-auth-enabled-requires-token-d4ed2c08/networking.istio.io/destinationrules/auth-enabled-test-kserve-workload-svc.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.120661639Z istio/namespaces/e2e-test-llm-inference-service-48639af5/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.120668549Z istio/namespaces/e2e-test-llm-inference-service-48639af5/gateway.networking.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.120672389Z istio/namespaces/e2e-test-llm-inference-service-48639af5/gateway.networking.k8s.io/httproutes/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.12072088Z istio/namespaces/e2e-test-llm-inference-service-48639af5/gateway.networking.k8s.io/httproutes/llmisvc-model-pvc-router-manage-e8706282-kserve-route.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.120790551Z istio/namespaces/e2e-test-llm-inference-service-48639af5/inference.networking.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.120799521Z istio/namespaces/e2e-test-llm-inference-service-48639af5/inference.networking.k8s.io/inferencepools/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.120847632Z istio/namespaces/e2e-test-llm-inference-service-48639af5/inference.networking.k8s.io/inferencepools/llmisvc-model-pvc-router-manage-e8706282-inference-pool.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.120899303Z istio/namespaces/e2e-test-llm-inference-service-48639af5/inference.networking.x-k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.120909084Z istio/namespaces/e2e-test-llm-inference-service-48639af5/inference.networking.x-k8s.io/inferencepools/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.120960825Z istio/namespaces/e2e-test-llm-inference-service-48639af5/inference.networking.x-k8s.io/inferencepools/llmisvc-model-pvc-router-manage-e8706282-inference-pool.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.121010756Z istio/namespaces/e2e-test-llm-inference-service-48639af5/networking.istio.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.121024286Z istio/namespaces/e2e-test-llm-inference-service-48639af5/networking.istio.io/destinationrules/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.121090807Z istio/namespaces/e2e-test-llm-inference-service-48639af5/networking.istio.io/destinationrules/llmisvc-model-pvc-router-manage-e8706282-kserve-scheduler.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.121206719Z istio/namespaces/e2e-test-llm-inference-service-48639af5/networking.istio.io/destinationrules/llmisvc-model-pvc-router-manage-e8706282-kserve-shadow-svc.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.121363633Z istio/namespaces/e2e-test-llm-inference-service-48639af5/networking.istio.io/destinationrules/llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.121449975Z istio/namespaces/e2e-test-llm-inference-service-fac1c4ae/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.121464295Z istio/namespaces/e2e-test-llm-inference-service-fac1c4ae/gateway.networking.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.121468375Z istio/namespaces/e2e-test-llm-inference-service-fac1c4ae/gateway.networking.k8s.io/httproutes/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.121497645Z istio/namespaces/e2e-test-llm-inference-service-fac1c4ae/gateway.networking.k8s.io/httproutes/llmisvc-model-fb-opt-125m-route-3c960099-kserve-route.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.121586337Z istio/namespaces/e2e-test-llm-inference-service-fac1c4ae/networking.istio.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.121593787Z istio/namespaces/e2e-test-llm-inference-service-fac1c4ae/networking.istio.io/destinationrules/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.121639658Z istio/namespaces/e2e-test-llm-inference-service-fac1c4ae/networking.istio.io/destinationrules/llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.12174811Z istio/namespaces/openshift-ingress/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.121796571Z istio/namespaces/openshift-ingress/openshift-ingress.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.121874413Z istio/namespaces/openshift-ingress/apps.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.121918564Z istio/namespaces/openshift-ingress/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.122005526Z istio/namespaces/openshift-ingress/apps/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.122028756Z istio/namespaces/openshift-ingress/apps/daemonsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.122137728Z istio/namespaces/openshift-ingress/apps/deployments.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.122378633Z istio/namespaces/openshift-ingress/apps/replicasets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.122588587Z istio/namespaces/openshift-ingress/apps/statefulsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.122651869Z istio/namespaces/openshift-ingress/autoscaling/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.122681649Z istio/namespaces/openshift-ingress/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.122756301Z istio/namespaces/openshift-ingress/batch/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.122818572Z istio/namespaces/openshift-ingress/batch/cronjobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.122887654Z istio/namespaces/openshift-ingress/batch/jobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.122955015Z istio/namespaces/openshift-ingress/build.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.122984816Z istio/namespaces/openshift-ingress/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.123077597Z istio/namespaces/openshift-ingress/build.openshift.io/builds.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.12317157Z istio/namespaces/openshift-ingress/core/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.12320825Z istio/namespaces/openshift-ingress/core/configmaps.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.123637779Z istio/namespaces/openshift-ingress/core/endpoints.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.123733921Z istio/namespaces/openshift-ingress/core/events.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.123962626Z istio/namespaces/openshift-ingress/core/persistentvolumeclaims.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.124059417Z istio/namespaces/openshift-ingress/core/pods.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.124359894Z istio/namespaces/openshift-ingress/core/replicationcontrollers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.124525267Z istio/namespaces/openshift-ingress/core/secrets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.124758512Z istio/namespaces/openshift-ingress/core/services.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.124858234Z istio/namespaces/openshift-ingress/discovery.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.124892084Z istio/namespaces/openshift-ingress/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.124989486Z istio/namespaces/openshift-ingress/gateway.networking.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.124996587Z istio/namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.125024907Z istio/namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/openshift-ai-inference.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.125107379Z istio/namespaces/openshift-ingress/image.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.125246172Z istio/namespaces/openshift-ingress/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.125317423Z istio/namespaces/openshift-ingress/k8s.ovn.org/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.125354394Z istio/namespaces/openshift-ingress/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.125449996Z istio/namespaces/openshift-ingress/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.125563188Z istio/namespaces/openshift-ingress/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.125817333Z istio/namespaces/openshift-ingress/monitoring.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.125851914Z istio/namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.125905445Z istio/namespaces/openshift-ingress/networking.istio.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.125931766Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.125975306Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/kuadrant-auth-openshift-ai-inference.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.126077498Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/kuadrant-openshift-ai-inference.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.126281253Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/openshift-ai-inference-authn-ssl.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.126392295Z istio/namespaces/openshift-ingress/networking.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.126425936Z istio/namespaces/openshift-ingress/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.126498097Z istio/namespaces/openshift-ingress/pods/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.126505037Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-bv8fm/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.126550008Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-bv8fm/istiod-openshift-gateway-5677ccb49b-bv8fm.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.126708351Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-bv8fm/discovery/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.126717451Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-bv8fm/discovery/discovery/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.126722231Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-bv8fm/discovery/discovery/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.126773032Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-bv8fm/discovery/discovery/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.137383358Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-bv8fm/discovery/discovery/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.137457589Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-bv8fm/discovery/discovery/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.137523881Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.137541981Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.137678404Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l/istio-proxy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.137688914Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l/istio-proxy/istio-proxy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.137697154Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.137751925Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.139054502Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.139138034Z istio/namespaces/openshift-ingress/pods/router-default-66dc97f78c-hsl5f/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.139206735Z istio/namespaces/openshift-ingress/pods/router-default-66dc97f78c-hsl5f/router-default-66dc97f78c-hsl5f.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.139316527Z istio/namespaces/openshift-ingress/pods/router-default-66dc97f78c-hsl5f/router/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.139328477Z istio/namespaces/openshift-ingress/pods/router-default-66dc97f78c-hsl5f/router/router/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.139339098Z istio/namespaces/openshift-ingress/pods/router-default-66dc97f78c-hsl5f/router/router/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.139375308Z istio/namespaces/openshift-ingress/pods/router-default-66dc97f78c-hsl5f/router/router/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.139483981Z istio/namespaces/openshift-ingress/pods/router-default-66dc97f78c-hsl5f/router/router/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.139553812Z istio/namespaces/openshift-ingress/pods/router-default-66dc97f78c-hsl5f/router/router/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.139597813Z istio/namespaces/openshift-ingress/policy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.139628523Z istio/namespaces/openshift-ingress/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.139686374Z istio/namespaces/openshift-ingress/route.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.139722645Z istio/namespaces/openshift-ingress/route.openshift.io/routes.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.139777716Z machine_config_ondisk/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.139784636Z machine_config_termination_logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.139789097Z monitoring/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.139793097Z monitoring/alertmanager/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.139829348Z monitoring/alertmanager/status.json [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.139917259Z monitoring/alertmanager/status.stderr [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.13995071Z monitoring/prometheus/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.1399844Z monitoring/prometheus/alertmanagers.json [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.140087953Z monitoring/prometheus/alertmanagers.stderr [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.140217575Z monitoring/prometheus/rules.json [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.141001471Z monitoring/prometheus/rules.stderr [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.141014412Z monitoring/prometheus/prometheus-k8s-0/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.141088733Z monitoring/prometheus/prometheus-k8s-0/active-targets.json [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.14191906Z monitoring/prometheus/prometheus-k8s-0/active-targets.stderr [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.141950091Z monitoring/prometheus/prometheus-k8s-0/status/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.142010592Z monitoring/prometheus/prometheus-k8s-0/status/runtimeinfo.json [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.142113514Z monitoring/prometheus/prometheus-k8s-0/status/runtimeinfo.stderr [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.142777397Z monitoring/prometheus/prometheus-k8s-0/status/tsdb.json [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.142870209Z monitoring/prometheus/prometheus-k8s-0/status/tsdb.stderr [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.14293272Z monitoring/prometheus/status/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.142957121Z monitoring/prometheus/status/config.json [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.143598074Z monitoring/prometheus/status/config.stderr [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.143668995Z monitoring/prometheus/status/flags.json [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.143753487Z monitoring/prometheus/status/flags.stderr [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.144005982Z namespaces/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.144013773Z namespaces/cert-manager-operator/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.144018013Z namespaces/cert-manager-operator/coordination.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.144031413Z namespaces/cert-manager-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.144072134Z namespaces/cert-manager-operator/coordination.k8s.io/leases/cert-manager-operator-lock.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.144208196Z namespaces/cert-manager-operator/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.144222097Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.144285298Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.144512783Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.144720947Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.14486878Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.145096594Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.14535238Z namespaces/cert-manager-operator/operators.coreos.com/installplans/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.14538628Z namespaces/cert-manager-operator/operators.coreos.com/installplans/install-z4979.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.145526363Z namespaces/cert-manager-operator/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.145561164Z namespaces/cert-manager-operator/operators.coreos.com/operatorconditions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.145623975Z namespaces/cert-manager-operator/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.145658516Z namespaces/cert-manager-operator/operators.coreos.com/operatorgroups/openshift-cert-manager-operator.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.145725907Z namespaces/cert-manager-operator/operators.coreos.com/subscriptions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.145784158Z namespaces/cert-manager-operator/operators.coreos.com/subscriptions/openshift-cert-manager-operator.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.14586344Z namespaces/cert-manager/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.14587593Z namespaces/cert-manager/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.145892851Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.145928091Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.146081404Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.146299609Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.146480532Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.146703807Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.146935302Z namespaces/default/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.146983033Z namespaces/default/default.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.147052254Z namespaces/default/apps.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.147093735Z namespaces/default/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.147212277Z namespaces/default/apps/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.147290709Z namespaces/default/apps/daemonsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.147380891Z namespaces/default/apps/deployments.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.147460212Z namespaces/default/apps/replicasets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.147540344Z namespaces/default/apps/statefulsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.147587035Z namespaces/default/autoscaling/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.147629766Z namespaces/default/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.147707407Z namespaces/default/batch/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.147734118Z namespaces/default/batch/cronjobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.14783469Z namespaces/default/batch/jobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.147892371Z namespaces/default/build.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.147945262Z namespaces/default/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.148070395Z namespaces/default/build.openshift.io/builds.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.148149066Z namespaces/default/core/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.148203357Z namespaces/default/core/configmaps.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.14831509Z namespaces/default/core/endpoints.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.148414812Z namespaces/default/core/events.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.148587235Z namespaces/default/core/persistentvolumeclaims.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.148656167Z namespaces/default/core/pods.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.148739048Z namespaces/default/core/replicationcontrollers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.148872591Z namespaces/default/core/secrets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.149054105Z namespaces/default/core/services.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.149149967Z namespaces/default/discovery.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.149198708Z namespaces/default/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.149266859Z namespaces/default/image.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.14931836Z namespaces/default/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.149357261Z namespaces/default/k8s.ovn.org/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.149399102Z namespaces/default/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.149479513Z namespaces/default/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.149555575Z namespaces/default/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.149602136Z namespaces/default/monitoring.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.149699428Z namespaces/default/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.149751949Z namespaces/default/networking.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.14980167Z namespaces/default/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.149845891Z namespaces/default/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.149854701Z namespaces/default/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.149882221Z namespaces/default/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.150048245Z namespaces/default/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.150237979Z namespaces/default/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.150373842Z namespaces/default/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.150574086Z namespaces/default/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.150763479Z namespaces/default/policy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.15080985Z namespaces/default/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.150863812Z namespaces/default/route.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.150909622Z namespaces/default/route.openshift.io/routes.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.150970104Z namespaces/e2e-test-llm-auth-enabled-requires-token-d4ed2c08/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.150991604Z namespaces/e2e-test-llm-auth-enabled-requires-token-d4ed2c08/monitoring.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.150996034Z namespaces/e2e-test-llm-auth-enabled-requires-token-d4ed2c08/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.151026695Z namespaces/e2e-test-llm-auth-enabled-requires-token-d4ed2c08/monitoring.coreos.com/podmonitors/kserve-llm-isvc-vllm-engine-default.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.151134947Z namespaces/e2e-test-llm-auth-enabled-requires-token-d4ed2c08/monitoring.coreos.com/podmonitors/kserve-llm-isvc-vllm-engine.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.151211048Z namespaces/e2e-test-llm-auth-enabled-requires-token-d4ed2c08/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.15126258Z namespaces/e2e-test-llm-auth-enabled-requires-token-d4ed2c08/monitoring.coreos.com/servicemonitors/kserve-llm-isvc-scheduler-default.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.151365302Z namespaces/e2e-test-llm-auth-enabled-requires-token-d4ed2c08/monitoring.coreos.com/servicemonitors/kserve-llm-isvc-scheduler.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.151429363Z namespaces/e2e-test-llm-auth-enabled-requires-token-d4ed2c08/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.151442053Z namespaces/e2e-test-llm-auth-enabled-requires-token-d4ed2c08/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.151489694Z namespaces/e2e-test-llm-auth-enabled-requires-token-d4ed2c08/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.151642777Z namespaces/e2e-test-llm-auth-enabled-requires-token-d4ed2c08/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.151816481Z namespaces/e2e-test-llm-auth-enabled-requires-token-d4ed2c08/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.151955914Z namespaces/e2e-test-llm-auth-enabled-requires-token-d4ed2c08/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.152236359Z namespaces/e2e-test-llm-auth-enabled-requires-token-d4ed2c08/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.152415773Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.152424143Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.152428313Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.152472104Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.152622667Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.152806761Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.152941183Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.153179458Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.153371332Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.153381992Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.153390323Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.153434113Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.153589287Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.153773391Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.153896543Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.154097207Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.154325182Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.154343042Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.154349342Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.154414653Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.154575457Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.154765571Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.154897843Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.155116108Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.155506326Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.155520096Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.155528356Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.155569207Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.15572435Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.155902254Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.156037266Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.156260651Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.156464205Z namespaces/e2e-test-llm-inference-service-48639af5/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.156475595Z namespaces/e2e-test-llm-inference-service-48639af5/monitoring.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.156481226Z namespaces/e2e-test-llm-inference-service-48639af5/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.156523486Z namespaces/e2e-test-llm-inference-service-48639af5/monitoring.coreos.com/podmonitors/kserve-llm-isvc-vllm-engine-default.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.156630429Z namespaces/e2e-test-llm-inference-service-48639af5/monitoring.coreos.com/podmonitors/kserve-llm-isvc-vllm-engine.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.15668243Z namespaces/e2e-test-llm-inference-service-48639af5/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.15671847Z namespaces/e2e-test-llm-inference-service-48639af5/monitoring.coreos.com/servicemonitors/kserve-llm-isvc-scheduler-default.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.156804842Z namespaces/e2e-test-llm-inference-service-48639af5/monitoring.coreos.com/servicemonitors/kserve-llm-isvc-scheduler.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.156873934Z namespaces/e2e-test-llm-inference-service-48639af5/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.156889164Z namespaces/e2e-test-llm-inference-service-48639af5/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.156927684Z namespaces/e2e-test-llm-inference-service-48639af5/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.157118498Z namespaces/e2e-test-llm-inference-service-48639af5/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.157336753Z namespaces/e2e-test-llm-inference-service-48639af5/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.157480986Z namespaces/e2e-test-llm-inference-service-48639af5/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.15769078Z namespaces/e2e-test-llm-inference-service-48639af5/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.157874664Z namespaces/e2e-test-llm-inference-service-fac1c4ae/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.157886644Z namespaces/e2e-test-llm-inference-service-fac1c4ae/monitoring.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.157890874Z namespaces/e2e-test-llm-inference-service-fac1c4ae/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.157897964Z namespaces/e2e-test-llm-inference-service-fac1c4ae/monitoring.coreos.com/podmonitors/kserve-llm-isvc-vllm-engine-default.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.158004596Z namespaces/e2e-test-llm-inference-service-fac1c4ae/monitoring.coreos.com/podmonitors/kserve-llm-isvc-vllm-engine.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.158054217Z namespaces/e2e-test-llm-inference-service-fac1c4ae/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.158102258Z namespaces/e2e-test-llm-inference-service-fac1c4ae/monitoring.coreos.com/servicemonitors/kserve-llm-isvc-scheduler-default.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.158214501Z namespaces/e2e-test-llm-inference-service-fac1c4ae/monitoring.coreos.com/servicemonitors/kserve-llm-isvc-scheduler.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.158262671Z namespaces/e2e-test-llm-inference-service-fac1c4ae/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.158278442Z namespaces/e2e-test-llm-inference-service-fac1c4ae/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.158334023Z namespaces/e2e-test-llm-inference-service-fac1c4ae/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.158481956Z namespaces/e2e-test-llm-inference-service-fac1c4ae/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.15870688Z namespaces/e2e-test-llm-inference-service-fac1c4ae/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.158810913Z namespaces/e2e-test-llm-inference-service-fac1c4ae/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.159013087Z namespaces/e2e-test-llm-inference-service-fac1c4ae/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.159214921Z namespaces/kserve-ci-e2e-test/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.159225081Z namespaces/kserve-ci-e2e-test/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.159232741Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.159287452Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.159439075Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.159611789Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.159753352Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.159972586Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.160219211Z namespaces/kserve/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.160264592Z namespaces/kserve/kserve.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.160366674Z namespaces/kserve/apps.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.160388565Z namespaces/kserve/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.160470516Z namespaces/kserve/apps/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.160499217Z namespaces/kserve/apps/daemonsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.160590229Z namespaces/kserve/apps/deployments.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.160812733Z namespaces/kserve/apps/replicasets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.160992007Z namespaces/kserve/apps/statefulsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.161033828Z namespaces/kserve/autoscaling/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.161086719Z namespaces/kserve/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.161211381Z namespaces/kserve/batch/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.161269872Z namespaces/kserve/batch/cronjobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.161406965Z namespaces/kserve/batch/jobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.161493057Z namespaces/kserve/build.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.161543618Z namespaces/kserve/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.16163212Z namespaces/kserve/build.openshift.io/builds.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.161666191Z namespaces/kserve/coordination.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.161670761Z namespaces/kserve/coordination.k8s.io/leases/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.161717202Z namespaces/kserve/coordination.k8s.io/leases/kserve-controller-manager-leader-lock.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.161799543Z namespaces/kserve/coordination.k8s.io/leases/llminferenceservice-kserve-controller-manager.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.161885655Z namespaces/kserve/coordination.k8s.io/leases/odh-model-controller.opendatahub.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.161983087Z namespaces/kserve/core/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.162028918Z namespaces/kserve/core/configmaps.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.162331184Z namespaces/kserve/core/endpoints.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.162438636Z namespaces/kserve/core/events.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.162880735Z namespaces/kserve/core/persistentvolumeclaims.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.162970147Z namespaces/kserve/core/pods.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.163290404Z namespaces/kserve/core/replicationcontrollers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.163414466Z namespaces/kserve/core/secrets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.163652361Z namespaces/kserve/core/services.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.163807724Z namespaces/kserve/discovery.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.163853045Z namespaces/kserve/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.163946947Z namespaces/kserve/image.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.163982018Z namespaces/kserve/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.164053999Z namespaces/kserve/k8s.ovn.org/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.16409777Z namespaces/kserve/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.164208392Z namespaces/kserve/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.164279524Z namespaces/kserve/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.164341555Z namespaces/kserve/monitoring.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.164359365Z namespaces/kserve/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.164431327Z namespaces/kserve/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.164482738Z namespaces/kserve/monitoring.coreos.com/servicemonitors/model-serving-api-metrics.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.16458369Z namespaces/kserve/monitoring.coreos.com/servicemonitors/odh-model-controller-metrics-monitor.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.164643531Z namespaces/kserve/networking.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.164671772Z namespaces/kserve/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.164734213Z namespaces/kserve/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.164744503Z namespaces/kserve/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.164804654Z namespaces/kserve/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.164965848Z namespaces/kserve/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.165148191Z namespaces/kserve/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.165308845Z namespaces/kserve/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.165528799Z namespaces/kserve/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.165712673Z namespaces/kserve/pods/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.165726743Z namespaces/kserve/pods/kserve-controller-manager-85d8b8ddfc-svxs4/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.165764264Z namespaces/kserve/pods/kserve-controller-manager-85d8b8ddfc-svxs4/kserve-controller-manager-85d8b8ddfc-svxs4.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.165858986Z namespaces/kserve/pods/kserve-controller-manager-85d8b8ddfc-svxs4/manager/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.165867756Z namespaces/kserve/pods/kserve-controller-manager-85d8b8ddfc-svxs4/manager/manager/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.165876876Z namespaces/kserve/pods/kserve-controller-manager-85d8b8ddfc-svxs4/manager/manager/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.165899726Z namespaces/kserve/pods/kserve-controller-manager-85d8b8ddfc-svxs4/manager/manager/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.166037059Z namespaces/kserve/pods/kserve-controller-manager-85d8b8ddfc-svxs4/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.166108631Z namespaces/kserve/pods/kserve-controller-manager-85d8b8ddfc-svxs4/manager/manager/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.166172882Z namespaces/kserve/pods/llmisvc-controller-manager-658796f68b-tpds6/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.166207973Z namespaces/kserve/pods/llmisvc-controller-manager-658796f68b-tpds6/llmisvc-controller-manager-658796f68b-tpds6.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.166307055Z namespaces/kserve/pods/llmisvc-controller-manager-658796f68b-tpds6/manager/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.166316295Z namespaces/kserve/pods/llmisvc-controller-manager-658796f68b-tpds6/manager/manager/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.166325025Z namespaces/kserve/pods/llmisvc-controller-manager-658796f68b-tpds6/manager/manager/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.166370826Z namespaces/kserve/pods/llmisvc-controller-manager-658796f68b-tpds6/manager/manager/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.170455689Z namespaces/kserve/pods/llmisvc-controller-manager-658796f68b-tpds6/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.170540551Z namespaces/kserve/pods/llmisvc-controller-manager-658796f68b-tpds6/manager/manager/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.170605202Z namespaces/kserve/pods/model-serving-api-7b954ffc95-phr94/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.170667883Z namespaces/kserve/pods/model-serving-api-7b954ffc95-phr94/model-serving-api-7b954ffc95-phr94.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.170749515Z namespaces/kserve/pods/model-serving-api-7b954ffc95-phr94/server/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.170759985Z namespaces/kserve/pods/model-serving-api-7b954ffc95-phr94/server/server/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.170766905Z namespaces/kserve/pods/model-serving-api-7b954ffc95-phr94/server/server/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.170811526Z namespaces/kserve/pods/model-serving-api-7b954ffc95-phr94/server/server/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.170903968Z namespaces/kserve/pods/model-serving-api-7b954ffc95-phr94/server/server/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.17098254Z namespaces/kserve/pods/model-serving-api-7b954ffc95-phr94/server/server/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.17102065Z namespaces/kserve/pods/odh-model-controller-75d9cdbddf-zxn8g/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.171047781Z namespaces/kserve/pods/odh-model-controller-75d9cdbddf-zxn8g/odh-model-controller-75d9cdbddf-zxn8g.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.171180344Z namespaces/kserve/pods/odh-model-controller-75d9cdbddf-zxn8g/manager/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.171197244Z namespaces/kserve/pods/odh-model-controller-75d9cdbddf-zxn8g/manager/manager/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.171208024Z namespaces/kserve/pods/odh-model-controller-75d9cdbddf-zxn8g/manager/manager/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.171236625Z namespaces/kserve/pods/odh-model-controller-75d9cdbddf-zxn8g/manager/manager/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.193452406Z namespaces/kserve/pods/odh-model-controller-75d9cdbddf-zxn8g/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.193535097Z namespaces/kserve/pods/odh-model-controller-75d9cdbddf-zxn8g/manager/manager/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.193559008Z namespaces/kserve/pods/s3-init-jlqc6/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.193613129Z namespaces/kserve/pods/s3-init-jlqc6/s3-init-jlqc6.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.193750172Z namespaces/kserve/pods/s3-init-jlqc6/download-hf-model/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.193760232Z namespaces/kserve/pods/s3-init-jlqc6/download-hf-model/download-hf-model/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.193765232Z namespaces/kserve/pods/s3-init-jlqc6/download-hf-model/download-hf-model/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.193803493Z namespaces/kserve/pods/s3-init-jlqc6/download-hf-model/download-hf-model/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.193912265Z namespaces/kserve/pods/s3-init-jlqc6/download-hf-model/download-hf-model/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.193998537Z namespaces/kserve/pods/s3-init-jlqc6/download-hf-model/download-hf-model/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.194047848Z namespaces/kserve/pods/s3-init-jlqc6/s3-init/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.194059018Z namespaces/kserve/pods/s3-init-jlqc6/s3-init/s3-init/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.194064028Z namespaces/kserve/pods/s3-init-jlqc6/s3-init/s3-init/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.194106659Z namespaces/kserve/pods/s3-init-jlqc6/s3-init/s3-init/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.194454546Z namespaces/kserve/pods/s3-init-jlqc6/s3-init/s3-init/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.194530818Z namespaces/kserve/pods/s3-init-jlqc6/s3-init/s3-init/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.194569668Z namespaces/kserve/pods/seaweedfs-64568bcd49-msmzj/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.19463148Z namespaces/kserve/pods/seaweedfs-64568bcd49-msmzj/seaweedfs-64568bcd49-msmzj.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.194722442Z namespaces/kserve/pods/seaweedfs-64568bcd49-msmzj/seaweedfs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.194733202Z namespaces/kserve/pods/seaweedfs-64568bcd49-msmzj/seaweedfs/seaweedfs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.194740982Z namespaces/kserve/pods/seaweedfs-64568bcd49-msmzj/seaweedfs/seaweedfs/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.194781763Z namespaces/kserve/pods/seaweedfs-64568bcd49-msmzj/seaweedfs/seaweedfs/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.195010427Z namespaces/kserve/pods/seaweedfs-64568bcd49-msmzj/seaweedfs/seaweedfs/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.195098719Z namespaces/kserve/pods/seaweedfs-64568bcd49-msmzj/seaweedfs/seaweedfs/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.195170211Z namespaces/kserve/policy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.195207291Z namespaces/kserve/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.195261642Z namespaces/kserve/route.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.195302573Z namespaces/kserve/route.openshift.io/routes.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.195402095Z namespaces/kuadrant-system/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.195459997Z namespaces/kuadrant-system/kuadrant-system.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.195538058Z namespaces/kuadrant-system/apps.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.195594879Z namespaces/kuadrant-system/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.195670271Z namespaces/kuadrant-system/apps/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.195720782Z namespaces/kuadrant-system/apps/daemonsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.195836984Z namespaces/kuadrant-system/apps/deployments.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.196185441Z namespaces/kuadrant-system/apps/replicasets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.196477237Z namespaces/kuadrant-system/apps/statefulsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.196533238Z namespaces/kuadrant-system/autoscaling/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.19659582Z namespaces/kuadrant-system/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.196655311Z namespaces/kuadrant-system/batch/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.196715362Z namespaces/kuadrant-system/batch/cronjobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.196825014Z namespaces/kuadrant-system/batch/jobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.196875075Z namespaces/kuadrant-system/build.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.196920766Z namespaces/kuadrant-system/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.197020328Z namespaces/kuadrant-system/build.openshift.io/builds.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.197062139Z namespaces/kuadrant-system/coordination.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.19711319Z namespaces/kuadrant-system/coordination.k8s.io/leases/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.197190512Z namespaces/kuadrant-system/coordination.k8s.io/leases/3745a16e.kuadrant.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.197295294Z namespaces/kuadrant-system/coordination.k8s.io/leases/a3f98d6c.kuadrant.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.197405796Z namespaces/kuadrant-system/coordination.k8s.io/leases/aac3a15d.authorino.kuadrant.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.197491448Z namespaces/kuadrant-system/coordination.k8s.io/leases/f139389e.kuadrant.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.197558719Z namespaces/kuadrant-system/core/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.19761493Z namespaces/kuadrant-system/core/configmaps.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.197841865Z namespaces/kuadrant-system/core/endpoints.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.197988758Z namespaces/kuadrant-system/core/events.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.198459137Z namespaces/kuadrant-system/core/persistentvolumeclaims.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.198538149Z namespaces/kuadrant-system/core/pods.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.198927117Z namespaces/kuadrant-system/core/replicationcontrollers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.19907751Z namespaces/kuadrant-system/core/secrets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.199357065Z namespaces/kuadrant-system/core/services.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.199480988Z namespaces/kuadrant-system/discovery.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.199527189Z namespaces/kuadrant-system/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.199655742Z namespaces/kuadrant-system/image.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.199720833Z namespaces/kuadrant-system/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.199769604Z namespaces/kuadrant-system/k8s.ovn.org/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.199816385Z namespaces/kuadrant-system/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.199919087Z namespaces/kuadrant-system/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.200010719Z namespaces/kuadrant-system/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.200053619Z namespaces/kuadrant-system/monitoring.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.200103111Z namespaces/kuadrant-system/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.200179832Z namespaces/kuadrant-system/networking.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.200226293Z namespaces/kuadrant-system/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.200307165Z namespaces/kuadrant-system/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.200325915Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.200378276Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.200592081Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.200844966Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.201011989Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.201439048Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.201649902Z namespaces/kuadrant-system/operators.coreos.com/installplans/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.201707893Z namespaces/kuadrant-system/operators.coreos.com/installplans/install-4k5rm.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.20204928Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.202100681Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.202229664Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.202336306Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.202434748Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.202480849Z namespaces/kuadrant-system/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.2025222Z namespaces/kuadrant-system/operators.coreos.com/operatorgroups/kuadrant.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.202599301Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.202654372Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/authorino-operator-stable-redhat-operators-openshift-marketplace.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.202749154Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/dns-operator-stable-redhat-operators-openshift-marketplace.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.202941248Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/limitador-operator-stable-redhat-operators-openshift-marketplace.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.203056841Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/rhcl-operator.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.203180273Z namespaces/kuadrant-system/pods/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.203190073Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-85tfv/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.203236804Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-85tfv/authorino-7cf4765b85-85tfv.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.203347937Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-85tfv/authorino/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.203358967Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-85tfv/authorino/authorino/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.203371887Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-85tfv/authorino/authorino/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.203428978Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-85tfv/authorino/authorino/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.214031093Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-85tfv/authorino/authorino/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.214142885Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-85tfv/authorino/authorino/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.214201297Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-2kw2j/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.214258018Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-2kw2j/authorino-operator-56fb6f6856-2kw2j.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.21435455Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-2kw2j/manager/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.21436687Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-2kw2j/manager/manager/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.21437554Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-2kw2j/manager/manager/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.214452942Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-2kw2j/manager/manager/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.214736408Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-2kw2j/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.21484445Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-2kw2j/manager/manager/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.214908101Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-qq2zk/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.214955762Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-qq2zk/dns-operator-controller-manager-ccbdf686-qq2zk.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.215086225Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-qq2zk/manager/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.215095855Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-qq2zk/manager/manager/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.215104075Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-qq2zk/manager/manager/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.215198967Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-qq2zk/manager/manager/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.215297139Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-qq2zk/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.215372201Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-qq2zk/manager/manager/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.215431542Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-vrjg8/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.215464442Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-vrjg8/kuadrant-console-plugin-598f86468b-vrjg8.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.215549714Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-vrjg8/kuadrant-console-plugin/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.215556844Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-vrjg8/kuadrant-console-plugin/kuadrant-console-plugin/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.215573085Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-vrjg8/kuadrant-console-plugin/kuadrant-console-plugin/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.215614295Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-vrjg8/kuadrant-console-plugin/kuadrant-console-plugin/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.215677387Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-vrjg8/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.215752728Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-vrjg8/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.215786799Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-2xg2c/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.21582058Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-2xg2c/kuadrant-operator-controller-manager-66866f5fcc-2xg2c.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.215933412Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-2xg2c/manager/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.215940782Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-2xg2c/manager/manager/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.215945052Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-2xg2c/manager/manager/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.215978463Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-2xg2c/manager/manager/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.22470109Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-2xg2c/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.224754231Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-2xg2c/manager/manager/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.224807052Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-5xp74/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.224856013Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-5xp74/limitador-limitador-85d9db4d5d-5xp74.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.224967185Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-5xp74/limitador/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.224976485Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-5xp74/limitador/limitador/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.224982645Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-5xp74/limitador/limitador/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.225030397Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-5xp74/limitador/limitador/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.225141989Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-5xp74/limitador/limitador/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.225249941Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-5xp74/limitador/limitador/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.225298672Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-xhtkz/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.225362263Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-xhtkz/limitador-operator-controller-manager-774d94476f-xhtkz.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.225448105Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-xhtkz/manager/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.225457025Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-xhtkz/manager/manager/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.225461965Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-xhtkz/manager/manager/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.225507806Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-xhtkz/manager/manager/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.22570396Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-xhtkz/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.225809922Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-xhtkz/manager/manager/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.225875014Z namespaces/kuadrant-system/policy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.225940155Z namespaces/kuadrant-system/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.226035867Z namespaces/kuadrant-system/route.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.226055517Z namespaces/kuadrant-system/route.openshift.io/routes.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.226114068Z namespaces/kube-node-lease/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.226138759Z namespaces/kube-node-lease/coordination.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.226152079Z namespaces/kube-node-lease/coordination.k8s.io/leases/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.22621643Z namespaces/kube-node-lease/coordination.k8s.io/leases/ip-10-0-135-188.ec2.internal.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.226312843Z namespaces/kube-node-lease/coordination.k8s.io/leases/ip-10-0-135-20.ec2.internal.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.226403894Z namespaces/kube-node-lease/coordination.k8s.io/leases/ip-10-0-140-52.ec2.internal.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.226461286Z namespaces/kube-node-lease/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.226471016Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.226524817Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.226762482Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.226970236Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.227137729Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.227370144Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.227583108Z namespaces/kube-public/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.227593809Z namespaces/kube-public/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.22764403Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.22769512Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.227896325Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.228073628Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.228240982Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.228492897Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.228775722Z namespaces/kube-system/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.228825703Z namespaces/kube-system/kube-system.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.228898485Z namespaces/kube-system/apps.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.228940416Z namespaces/kube-system/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.229011397Z namespaces/kube-system/apps/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.229073168Z namespaces/kube-system/apps/daemonsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.229222032Z namespaces/kube-system/apps/deployments.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.229306163Z namespaces/kube-system/apps/replicasets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.229392585Z namespaces/kube-system/apps/statefulsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.229439616Z namespaces/kube-system/autoscaling/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.229483917Z namespaces/kube-system/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.229550508Z namespaces/kube-system/batch/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.229603489Z namespaces/kube-system/batch/cronjobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.229673991Z namespaces/kube-system/batch/jobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.229711852Z namespaces/kube-system/build.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.229771773Z namespaces/kube-system/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.229853964Z namespaces/kube-system/build.openshift.io/builds.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.229899105Z namespaces/kube-system/coordination.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.229905525Z namespaces/kube-system/coordination.k8s.io/leases/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.230009938Z namespaces/kube-system/coordination.k8s.io/leases/apiserver-uamg2rs5l3kskmtxoiupylv5se.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.230093179Z namespaces/kube-system/coordination.k8s.io/leases/cert-manager-cainjector-leader-election.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.230223352Z namespaces/kube-system/coordination.k8s.io/leases/cert-manager-controller.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.230297943Z namespaces/kube-system/coordination.k8s.io/leases/kube-controller-manager.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.230477437Z namespaces/kube-system/coordination.k8s.io/leases/kube-scheduler.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.230544358Z namespaces/kube-system/core/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.23061923Z namespaces/kube-system/core/configmaps.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.230804974Z namespaces/kube-system/core/endpoints.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.230900336Z namespaces/kube-system/core/events.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.231320664Z namespaces/kube-system/core/persistentvolumeclaims.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.231392646Z namespaces/kube-system/core/pods.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.231658821Z namespaces/kube-system/core/replicationcontrollers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.231788074Z namespaces/kube-system/core/secrets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.23210599Z namespaces/kube-system/core/services.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.232188372Z namespaces/kube-system/discovery.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.232209952Z namespaces/kube-system/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.232282594Z namespaces/kube-system/image.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.232330375Z namespaces/kube-system/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.232409486Z namespaces/kube-system/k8s.ovn.org/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.232489068Z namespaces/kube-system/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.232573479Z namespaces/kube-system/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.232653811Z namespaces/kube-system/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.232702752Z namespaces/kube-system/monitoring.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.232753983Z namespaces/kube-system/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.232805104Z namespaces/kube-system/networking.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.232842675Z namespaces/kube-system/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.232899626Z namespaces/kube-system/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.232906616Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.232947127Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.23310254Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.233322155Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.233472108Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.233704682Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.233925137Z namespaces/kube-system/pods/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.233932707Z namespaces/kube-system/pods/global-pull-secret-syncer-27fcp/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.233993359Z namespaces/kube-system/pods/global-pull-secret-syncer-27fcp/global-pull-secret-syncer-27fcp.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.234098421Z namespaces/kube-system/pods/global-pull-secret-syncer-27fcp/global-pull-secret-syncer/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.234105821Z namespaces/kube-system/pods/global-pull-secret-syncer-27fcp/global-pull-secret-syncer/global-pull-secret-syncer/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.234109761Z namespaces/kube-system/pods/global-pull-secret-syncer-27fcp/global-pull-secret-syncer/global-pull-secret-syncer/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.234197763Z namespaces/kube-system/pods/global-pull-secret-syncer-27fcp/global-pull-secret-syncer/global-pull-secret-syncer/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.234618081Z namespaces/kube-system/pods/global-pull-secret-syncer-27fcp/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.234717633Z namespaces/kube-system/pods/global-pull-secret-syncer-27fcp/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.234778984Z namespaces/kube-system/pods/global-pull-secret-syncer-8sv5v/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.234846836Z namespaces/kube-system/pods/global-pull-secret-syncer-8sv5v/global-pull-secret-syncer-8sv5v.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.234920627Z namespaces/kube-system/pods/global-pull-secret-syncer-8sv5v/global-pull-secret-syncer/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.234929938Z namespaces/kube-system/pods/global-pull-secret-syncer-8sv5v/global-pull-secret-syncer/global-pull-secret-syncer/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.234934938Z namespaces/kube-system/pods/global-pull-secret-syncer-8sv5v/global-pull-secret-syncer/global-pull-secret-syncer/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.234991459Z namespaces/kube-system/pods/global-pull-secret-syncer-8sv5v/global-pull-secret-syncer/global-pull-secret-syncer/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.235419987Z namespaces/kube-system/pods/global-pull-secret-syncer-8sv5v/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.235502759Z namespaces/kube-system/pods/global-pull-secret-syncer-8sv5v/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.23555222Z namespaces/kube-system/pods/global-pull-secret-syncer-9l299/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.235596851Z namespaces/kube-system/pods/global-pull-secret-syncer-9l299/global-pull-secret-syncer-9l299.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.235685103Z namespaces/kube-system/pods/global-pull-secret-syncer-9l299/global-pull-secret-syncer/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.235694753Z namespaces/kube-system/pods/global-pull-secret-syncer-9l299/global-pull-secret-syncer/global-pull-secret-syncer/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.235699593Z namespaces/kube-system/pods/global-pull-secret-syncer-9l299/global-pull-secret-syncer/global-pull-secret-syncer/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.235714393Z namespaces/kube-system/pods/global-pull-secret-syncer-9l299/global-pull-secret-syncer/global-pull-secret-syncer/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.236096371Z namespaces/kube-system/pods/global-pull-secret-syncer-9l299/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.236199083Z namespaces/kube-system/pods/global-pull-secret-syncer-9l299/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.236244374Z namespaces/kube-system/pods/konnectivity-agent-b74hg/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.236293275Z namespaces/kube-system/pods/konnectivity-agent-b74hg/konnectivity-agent-b74hg.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.236370847Z namespaces/kube-system/pods/konnectivity-agent-b74hg/konnectivity-agent/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.236388277Z namespaces/kube-system/pods/konnectivity-agent-b74hg/konnectivity-agent/konnectivity-agent/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.236398617Z namespaces/kube-system/pods/konnectivity-agent-b74hg/konnectivity-agent/konnectivity-agent/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.236457558Z namespaces/kube-system/pods/konnectivity-agent-b74hg/konnectivity-agent/konnectivity-agent/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.237815056Z namespaces/kube-system/pods/konnectivity-agent-b74hg/konnectivity-agent/konnectivity-agent/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.237895978Z namespaces/kube-system/pods/konnectivity-agent-b74hg/konnectivity-agent/konnectivity-agent/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.237944868Z namespaces/kube-system/pods/konnectivity-agent-fz847/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.23799446Z namespaces/kube-system/pods/konnectivity-agent-fz847/konnectivity-agent-fz847.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.238076551Z namespaces/kube-system/pods/konnectivity-agent-fz847/konnectivity-agent/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.238084801Z namespaces/kube-system/pods/konnectivity-agent-fz847/konnectivity-agent/konnectivity-agent/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.238092551Z namespaces/kube-system/pods/konnectivity-agent-fz847/konnectivity-agent/konnectivity-agent/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.238138992Z namespaces/kube-system/pods/konnectivity-agent-fz847/konnectivity-agent/konnectivity-agent/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.239269555Z namespaces/kube-system/pods/konnectivity-agent-fz847/konnectivity-agent/konnectivity-agent/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.239314236Z namespaces/kube-system/pods/konnectivity-agent-fz847/konnectivity-agent/konnectivity-agent/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.239381628Z namespaces/kube-system/pods/konnectivity-agent-vvdjg/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.239423109Z namespaces/kube-system/pods/konnectivity-agent-vvdjg/konnectivity-agent-vvdjg.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.23949987Z namespaces/kube-system/pods/konnectivity-agent-vvdjg/konnectivity-agent/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.23951023Z namespaces/kube-system/pods/konnectivity-agent-vvdjg/konnectivity-agent/konnectivity-agent/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.23951796Z namespaces/kube-system/pods/konnectivity-agent-vvdjg/konnectivity-agent/konnectivity-agent/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.239526371Z namespaces/kube-system/pods/konnectivity-agent-vvdjg/konnectivity-agent/konnectivity-agent/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.240875238Z namespaces/kube-system/pods/konnectivity-agent-vvdjg/konnectivity-agent/konnectivity-agent/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.24096664Z namespaces/kube-system/pods/konnectivity-agent-vvdjg/konnectivity-agent/konnectivity-agent/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.240996401Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-135-188.ec2.internal/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.241055952Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-135-188.ec2.internal/kube-apiserver-proxy-ip-10-0-135-188.ec2.internal.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.241178484Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-135-188.ec2.internal/haproxy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.241190825Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-135-188.ec2.internal/haproxy/haproxy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.241203325Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-135-188.ec2.internal/haproxy/haproxy/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.241244966Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-135-188.ec2.internal/haproxy/haproxy/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.241574532Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-135-188.ec2.internal/haproxy/haproxy/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.241660204Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-135-188.ec2.internal/haproxy/haproxy/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.241723535Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-135-20.ec2.internal/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.241780386Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-135-20.ec2.internal/kube-apiserver-proxy-ip-10-0-135-20.ec2.internal.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.241857708Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-135-20.ec2.internal/haproxy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.241867388Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-135-20.ec2.internal/haproxy/haproxy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.241871608Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-135-20.ec2.internal/haproxy/haproxy/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.241920729Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-135-20.ec2.internal/haproxy/haproxy/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.242195125Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-135-20.ec2.internal/haproxy/haproxy/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.242271207Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-135-20.ec2.internal/haproxy/haproxy/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.242324697Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-140-52.ec2.internal/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.242382429Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-140-52.ec2.internal/kube-apiserver-proxy-ip-10-0-140-52.ec2.internal.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.24247354Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-140-52.ec2.internal/haproxy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.242485951Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-140-52.ec2.internal/haproxy/haproxy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.242492131Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-140-52.ec2.internal/haproxy/haproxy/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.242544112Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-140-52.ec2.internal/haproxy/haproxy/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.242878809Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-140-52.ec2.internal/haproxy/haproxy/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.24297045Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-140-52.ec2.internal/haproxy/haproxy/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.243007211Z namespaces/kube-system/policy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.243051142Z namespaces/kube-system/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.243110483Z namespaces/kube-system/rbac.authorization.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.243118144Z namespaces/kube-system/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.243188065Z namespaces/kube-system/rbac.authorization.k8s.io/rolebindings/csi-snapshot-controller-operator-authentication-reader.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.243302687Z namespaces/kube-system/rbac.authorization.k8s.io/rolebindings/network-diagnostics.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.243364418Z namespaces/kube-system/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.24341852Z namespaces/kube-system/rbac.authorization.k8s.io/roles/extension-apiserver-authentication-reader.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.243478121Z namespaces/kube-system/route.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.243521862Z namespaces/kube-system/route.openshift.io/routes.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.243577873Z namespaces/observability/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.243584733Z namespaces/observability/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.243591103Z namespaces/observability/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.243669275Z namespaces/observability/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.243830488Z namespaces/observability/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.244039212Z namespaces/observability/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.244225636Z namespaces/observability/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.244466461Z namespaces/observability/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.244763727Z namespaces/open-cluster-management-007a4926-ebb1-4604-8489-b69a2896c/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.244772337Z namespaces/open-cluster-management-007a4926-ebb1-4604-8489-b69a2896c/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.244780447Z namespaces/open-cluster-management-007a4926-ebb1-4604-8489-b69a2896c/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.244843608Z namespaces/open-cluster-management-007a4926-ebb1-4604-8489-b69a2896c/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.245057043Z namespaces/open-cluster-management-007a4926-ebb1-4604-8489-b69a2896c/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.245338359Z namespaces/open-cluster-management-007a4926-ebb1-4604-8489-b69a2896c/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.245489582Z namespaces/open-cluster-management-007a4926-ebb1-4604-8489-b69a2896c/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.245769137Z namespaces/open-cluster-management-007a4926-ebb1-4604-8489-b69a2896c/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.246000502Z namespaces/open-cluster-management-agent-addon/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.246009572Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.246014452Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/leases/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.246063443Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/leases/cluster-proxy.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.246197976Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/leases/work-manager.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.246259247Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.246274928Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.246339559Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.246513413Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.246737467Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.24690148Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.247179966Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.247602705Z namespaces/openshift-apiserver-operator/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.247658396Z namespaces/openshift-apiserver-operator/openshift-apiserver-operator.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.247749687Z namespaces/openshift-apiserver-operator/apps.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.247804709Z namespaces/openshift-apiserver-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.24787907Z namespaces/openshift-apiserver-operator/apps/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.247936221Z namespaces/openshift-apiserver-operator/apps/daemonsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.248055474Z namespaces/openshift-apiserver-operator/apps/deployments.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.248164016Z namespaces/openshift-apiserver-operator/apps/replicasets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.248259348Z namespaces/openshift-apiserver-operator/apps/statefulsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.248313549Z namespaces/openshift-apiserver-operator/autoscaling/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.24836258Z namespaces/openshift-apiserver-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.248422481Z namespaces/openshift-apiserver-operator/batch/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.248479982Z namespaces/openshift-apiserver-operator/batch/cronjobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.248569544Z namespaces/openshift-apiserver-operator/batch/jobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.248619795Z namespaces/openshift-apiserver-operator/build.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.248726027Z namespaces/openshift-apiserver-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.248811969Z namespaces/openshift-apiserver-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.24887558Z namespaces/openshift-apiserver-operator/core/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.248935251Z namespaces/openshift-apiserver-operator/core/configmaps.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.250189397Z namespaces/openshift-apiserver-operator/core/endpoints.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.250288529Z namespaces/openshift-apiserver-operator/core/events.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.250390011Z namespaces/openshift-apiserver-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.250512224Z namespaces/openshift-apiserver-operator/core/pods.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.250628346Z namespaces/openshift-apiserver-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.25080274Z namespaces/openshift-apiserver-operator/core/secrets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.251077375Z namespaces/openshift-apiserver-operator/core/services.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.251186417Z namespaces/openshift-apiserver-operator/discovery.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.251233728Z namespaces/openshift-apiserver-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.25129951Z namespaces/openshift-apiserver-operator/image.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.25134538Z namespaces/openshift-apiserver-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.251416872Z namespaces/openshift-apiserver-operator/k8s.ovn.org/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.251463263Z namespaces/openshift-apiserver-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.251575705Z namespaces/openshift-apiserver-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.251675957Z namespaces/openshift-apiserver-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.251760069Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.2518178Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.251899322Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.251949433Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/servicemonitors/openshift-apiserver-operator.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.252023754Z namespaces/openshift-apiserver-operator/networking.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.252069525Z namespaces/openshift-apiserver-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.252146367Z namespaces/openshift-apiserver-operator/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.252158427Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.252268489Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.252448153Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.252644717Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.2527997Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.253082146Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.25329895Z namespaces/openshift-apiserver-operator/policy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.253350641Z namespaces/openshift-apiserver-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.253405082Z namespaces/openshift-apiserver-operator/route.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.253444423Z namespaces/openshift-apiserver-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.253566696Z namespaces/openshift-apiserver/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.253606596Z namespaces/openshift-apiserver/openshift-apiserver.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.253721249Z namespaces/openshift-apiserver/apps.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.253742269Z namespaces/openshift-apiserver/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.253826191Z namespaces/openshift-apiserver/apps/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.253849171Z namespaces/openshift-apiserver/apps/daemonsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.253957984Z namespaces/openshift-apiserver/apps/deployments.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.254043155Z namespaces/openshift-apiserver/apps/replicasets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.254141997Z namespaces/openshift-apiserver/apps/statefulsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.254220189Z namespaces/openshift-apiserver/autoscaling/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.25426525Z namespaces/openshift-apiserver/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.254337871Z namespaces/openshift-apiserver/batch/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.254390892Z namespaces/openshift-apiserver/batch/cronjobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.254505395Z namespaces/openshift-apiserver/batch/jobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.254585386Z namespaces/openshift-apiserver/build.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.254665898Z namespaces/openshift-apiserver/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.25478539Z namespaces/openshift-apiserver/build.openshift.io/builds.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.255089627Z namespaces/openshift-apiserver/core/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.255166628Z namespaces/openshift-apiserver/core/configmaps.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.25528376Z namespaces/openshift-apiserver/core/endpoints.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.255387753Z namespaces/openshift-apiserver/core/events.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.255482835Z namespaces/openshift-apiserver/core/persistentvolumeclaims.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.255585157Z namespaces/openshift-apiserver/core/pods.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.255695199Z namespaces/openshift-apiserver/core/replicationcontrollers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.255854852Z namespaces/openshift-apiserver/core/secrets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.256079196Z namespaces/openshift-apiserver/core/services.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.256188979Z namespaces/openshift-apiserver/discovery.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.2562403Z namespaces/openshift-apiserver/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.256324912Z namespaces/openshift-apiserver/image.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.256364272Z namespaces/openshift-apiserver/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.256444464Z namespaces/openshift-apiserver/k8s.ovn.org/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.256500135Z namespaces/openshift-apiserver/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.256588387Z namespaces/openshift-apiserver/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.256686949Z namespaces/openshift-apiserver/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.2567595Z namespaces/openshift-apiserver/monitoring.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.256805601Z namespaces/openshift-apiserver/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.256885903Z namespaces/openshift-apiserver/networking.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.257019836Z namespaces/openshift-apiserver/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.257076587Z namespaces/openshift-apiserver/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.257101687Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.257209319Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.257392183Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.257592267Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.25773895Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.258092737Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.258303482Z namespaces/openshift-apiserver/policy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.258361673Z namespaces/openshift-apiserver/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.258431734Z namespaces/openshift-apiserver/route.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.258467075Z namespaces/openshift-apiserver/route.openshift.io/routes.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.258503896Z namespaces/openshift-authentication-operator/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.258514046Z namespaces/openshift-authentication-operator/monitoring.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.258530836Z namespaces/openshift-authentication-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.258605138Z namespaces/openshift-authentication-operator/monitoring.coreos.com/servicemonitors/authentication-operator.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.258664939Z namespaces/openshift-authentication-operator/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.258674589Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.258747741Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.258918434Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.259136949Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.259321932Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.259552587Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.259735141Z namespaces/openshift-authentication/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.259745731Z namespaces/openshift-authentication/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.259754731Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.259819743Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.259967015Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.260146319Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.260365804Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.260571508Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.260918895Z namespaces/openshift-cloud-controller-manager/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.260929205Z namespaces/openshift-cloud-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.260934865Z namespaces/openshift-cloud-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.260978086Z namespaces/openshift-cloud-controller-manager/coordination.k8s.io/leases/cloud-controller-manager.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.261053217Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.261065008Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.261105769Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.261287312Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.261460696Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.261605709Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.261817793Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.262010857Z namespaces/openshift-cloud-credential-operator/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.262018257Z namespaces/openshift-cloud-credential-operator/coordination.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.262023667Z namespaces/openshift-cloud-credential-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.262066838Z namespaces/openshift-cloud-credential-operator/coordination.k8s.io/leases/cloud-credential-operator-leader.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.262139819Z namespaces/openshift-cloud-credential-operator/monitoring.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.26215079Z namespaces/openshift-cloud-credential-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.262213081Z namespaces/openshift-cloud-credential-operator/monitoring.coreos.com/servicemonitors/cloud-credential-operator.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.262277152Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.262287722Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.262339654Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.262483126Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.26267209Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.262809523Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.263030018Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.263288273Z namespaces/openshift-cloud-network-config-controller/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.263372585Z namespaces/openshift-cloud-network-config-controller/openshift-cloud-network-config-controller.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.263433896Z namespaces/openshift-cloud-network-config-controller/apps.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.263454136Z namespaces/openshift-cloud-network-config-controller/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.263527538Z namespaces/openshift-cloud-network-config-controller/apps/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.263590269Z namespaces/openshift-cloud-network-config-controller/apps/daemonsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.263674741Z namespaces/openshift-cloud-network-config-controller/apps/deployments.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.263745742Z namespaces/openshift-cloud-network-config-controller/apps/replicasets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.263826114Z namespaces/openshift-cloud-network-config-controller/apps/statefulsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.263864254Z namespaces/openshift-cloud-network-config-controller/autoscaling/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.263909695Z namespaces/openshift-cloud-network-config-controller/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.263959996Z namespaces/openshift-cloud-network-config-controller/batch/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.264004687Z namespaces/openshift-cloud-network-config-controller/batch/cronjobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.2641398Z namespaces/openshift-cloud-network-config-controller/batch/jobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.264207961Z namespaces/openshift-cloud-network-config-controller/build.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.264226962Z namespaces/openshift-cloud-network-config-controller/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.264328634Z namespaces/openshift-cloud-network-config-controller/build.openshift.io/builds.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.264364575Z namespaces/openshift-cloud-network-config-controller/coordination.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.264373185Z namespaces/openshift-cloud-network-config-controller/coordination.k8s.io/leases/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.264423396Z namespaces/openshift-cloud-network-config-controller/coordination.k8s.io/leases/cloud-network-config-controller-lock.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.264500678Z namespaces/openshift-cloud-network-config-controller/core/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.264546428Z namespaces/openshift-cloud-network-config-controller/core/configmaps.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.264646861Z namespaces/openshift-cloud-network-config-controller/core/endpoints.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.264732212Z namespaces/openshift-cloud-network-config-controller/core/events.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.264818554Z namespaces/openshift-cloud-network-config-controller/core/persistentvolumeclaims.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.264899086Z namespaces/openshift-cloud-network-config-controller/core/pods.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.264979377Z namespaces/openshift-cloud-network-config-controller/core/replicationcontrollers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.2651403Z namespaces/openshift-cloud-network-config-controller/core/secrets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.265274393Z namespaces/openshift-cloud-network-config-controller/core/services.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.265322234Z namespaces/openshift-cloud-network-config-controller/core/serviceaccounts/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.265360305Z namespaces/openshift-cloud-network-config-controller/core/serviceaccounts/cloud-network-config-controller.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.265418416Z namespaces/openshift-cloud-network-config-controller/discovery.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.265451477Z namespaces/openshift-cloud-network-config-controller/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.265508048Z namespaces/openshift-cloud-network-config-controller/image.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.265535159Z namespaces/openshift-cloud-network-config-controller/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.26562299Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.265662891Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.265750093Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.265831094Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.265880595Z namespaces/openshift-cloud-network-config-controller/monitoring.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.265913586Z namespaces/openshift-cloud-network-config-controller/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.265968397Z namespaces/openshift-cloud-network-config-controller/networking.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.266014928Z namespaces/openshift-cloud-network-config-controller/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.266055059Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.266064829Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.26609334Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.266272183Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.266453407Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.26660111Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.266811744Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.266994808Z namespaces/openshift-cloud-network-config-controller/policy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.267033839Z namespaces/openshift-cloud-network-config-controller/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.26708947Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.26709686Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.267139801Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/rolebindings/cloud-network-config-controller-rb.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.267229023Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.267246513Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/roles/cloud-network-config-controller.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.267328485Z namespaces/openshift-cloud-network-config-controller/route.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.267383916Z namespaces/openshift-cloud-network-config-controller/route.openshift.io/routes.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.267474128Z namespaces/openshift-cluster-csi-drivers/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.267508819Z namespaces/openshift-cluster-csi-drivers/openshift-cluster-csi-drivers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.26757498Z namespaces/openshift-cluster-csi-drivers/apps.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.267614461Z namespaces/openshift-cluster-csi-drivers/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.267871366Z namespaces/openshift-cluster-csi-drivers/apps/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.267898356Z namespaces/openshift-cluster-csi-drivers/apps/daemonsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.268030789Z namespaces/openshift-cluster-csi-drivers/apps/deployments.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.268103081Z namespaces/openshift-cluster-csi-drivers/apps/replicasets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.268286244Z namespaces/openshift-cluster-csi-drivers/apps/statefulsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.268327565Z namespaces/openshift-cluster-csi-drivers/autoscaling/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.268357816Z namespaces/openshift-cluster-csi-drivers/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.268429957Z namespaces/openshift-cluster-csi-drivers/batch/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.268466138Z namespaces/openshift-cluster-csi-drivers/batch/cronjobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.26855045Z namespaces/openshift-cluster-csi-drivers/batch/jobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.268594561Z namespaces/openshift-cluster-csi-drivers/build.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.268647812Z namespaces/openshift-cluster-csi-drivers/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.268726283Z namespaces/openshift-cluster-csi-drivers/build.openshift.io/builds.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.268785805Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.268793054Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.268826225Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/ebs-csi-aws-com.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.268916087Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/external-attacher-leader-ebs-csi-aws-com.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.269001129Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/external-resizer-ebs-csi-aws-com.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.269173822Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/external-snapshotter-leader-ebs-csi-aws-com.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.269271144Z namespaces/openshift-cluster-csi-drivers/core/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.269317125Z namespaces/openshift-cluster-csi-drivers/core/configmaps.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.269421337Z namespaces/openshift-cluster-csi-drivers/core/endpoints.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.269504859Z namespaces/openshift-cluster-csi-drivers/core/events.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.269883517Z namespaces/openshift-cluster-csi-drivers/core/persistentvolumeclaims.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.269975838Z namespaces/openshift-cluster-csi-drivers/core/pods.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.270210963Z namespaces/openshift-cluster-csi-drivers/core/replicationcontrollers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.270371466Z namespaces/openshift-cluster-csi-drivers/core/secrets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.270488399Z namespaces/openshift-cluster-csi-drivers/core/services.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.27053514Z namespaces/openshift-cluster-csi-drivers/core/serviceaccounts/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.270578331Z namespaces/openshift-cluster-csi-drivers/core/serviceaccounts/aws-ebs-csi-driver-operator.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.270641572Z namespaces/openshift-cluster-csi-drivers/discovery.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.270679893Z namespaces/openshift-cluster-csi-drivers/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.270728074Z namespaces/openshift-cluster-csi-drivers/image.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.270766835Z namespaces/openshift-cluster-csi-drivers/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.270813146Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.270931048Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.27101814Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.271100312Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.271174903Z namespaces/openshift-cluster-csi-drivers/monitoring.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.271213554Z namespaces/openshift-cluster-csi-drivers/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.271265465Z namespaces/openshift-cluster-csi-drivers/networking.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.271304545Z namespaces/openshift-cluster-csi-drivers/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.271375217Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.271384627Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.271446879Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.271590951Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.271881457Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.27202031Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.272263995Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.272459429Z namespaces/openshift-cluster-csi-drivers/pods/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.272466579Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-9wttr/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.27252426Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-9wttr/aws-ebs-csi-driver-node-9wttr.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.272628792Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-9wttr/csi-driver/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.272635683Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-9wttr/csi-driver/csi-driver/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.272639713Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-9wttr/csi-driver/csi-driver/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.272689684Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-9wttr/csi-driver/csi-driver/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.272788536Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-9wttr/csi-driver/csi-driver/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.272864407Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-9wttr/csi-driver/csi-driver/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.272904908Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-9wttr/csi-liveness-probe/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.272911608Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-9wttr/csi-liveness-probe/csi-liveness-probe/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.272917868Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-9wttr/csi-liveness-probe/csi-liveness-probe/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.272961459Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-9wttr/csi-liveness-probe/csi-liveness-probe/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.273054321Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-9wttr/csi-liveness-probe/csi-liveness-probe/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.273141163Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-9wttr/csi-liveness-probe/csi-liveness-probe/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.273207734Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-9wttr/csi-node-driver-registrar/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.273215424Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-9wttr/csi-node-driver-registrar/csi-node-driver-registrar/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.273219535Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-9wttr/csi-node-driver-registrar/csi-node-driver-registrar/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.273268085Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-9wttr/csi-node-driver-registrar/csi-node-driver-registrar/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.273375988Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-9wttr/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.273442429Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-9wttr/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.27350147Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-bdlx6/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.273538181Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-bdlx6/aws-ebs-csi-driver-node-bdlx6.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.273626643Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-bdlx6/csi-driver/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.273633493Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-bdlx6/csi-driver/csi-driver/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.273639903Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-bdlx6/csi-driver/csi-driver/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.273710184Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-bdlx6/csi-driver/csi-driver/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.273801686Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-bdlx6/csi-driver/csi-driver/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.273870168Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-bdlx6/csi-driver/csi-driver/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.273906428Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-bdlx6/csi-liveness-probe/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.273914898Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-bdlx6/csi-liveness-probe/csi-liveness-probe/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.273924729Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-bdlx6/csi-liveness-probe/csi-liveness-probe/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.27398038Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-bdlx6/csi-liveness-probe/csi-liveness-probe/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.274066422Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-bdlx6/csi-liveness-probe/csi-liveness-probe/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.274141743Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-bdlx6/csi-liveness-probe/csi-liveness-probe/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.274196054Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-bdlx6/csi-node-driver-registrar/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.274206895Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-bdlx6/csi-node-driver-registrar/csi-node-driver-registrar/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.274213504Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-bdlx6/csi-node-driver-registrar/csi-node-driver-registrar/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.274256056Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-bdlx6/csi-node-driver-registrar/csi-node-driver-registrar/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.274356018Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-bdlx6/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.274425529Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-bdlx6/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.27448621Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wd5zb/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.274524011Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wd5zb/aws-ebs-csi-driver-node-wd5zb.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.274624673Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wd5zb/csi-driver/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.274714225Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wd5zb/csi-driver/csi-driver/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.274723865Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wd5zb/csi-driver/csi-driver/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.274776466Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wd5zb/csi-driver/csi-driver/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.274869788Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wd5zb/csi-driver/csi-driver/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.274939039Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wd5zb/csi-driver/csi-driver/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.27497741Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wd5zb/csi-liveness-probe/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.27498447Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wd5zb/csi-liveness-probe/csi-liveness-probe/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.27499073Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wd5zb/csi-liveness-probe/csi-liveness-probe/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.275049292Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wd5zb/csi-liveness-probe/csi-liveness-probe/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.275162964Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wd5zb/csi-liveness-probe/csi-liveness-probe/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.275333947Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wd5zb/csi-liveness-probe/csi-liveness-probe/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.275373618Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wd5zb/csi-node-driver-registrar/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.275380428Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wd5zb/csi-node-driver-registrar/csi-node-driver-registrar/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.275386728Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wd5zb/csi-node-driver-registrar/csi-node-driver-registrar/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.27545878Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wd5zb/csi-node-driver-registrar/csi-node-driver-registrar/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.275627073Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wd5zb/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.275696015Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wd5zb/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.275749716Z namespaces/openshift-cluster-csi-drivers/policy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.275789047Z namespaces/openshift-cluster-csi-drivers/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.275845868Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.275856708Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.275902049Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/rolebindings/aws-ebs-csi-driver-operator-rolebinding.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.27596501Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.276006431Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/roles/aws-ebs-csi-driver-operator-role.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.276072862Z namespaces/openshift-cluster-csi-drivers/route.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.276112473Z namespaces/openshift-cluster-csi-drivers/route.openshift.io/routes.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.276215695Z namespaces/openshift-cluster-machine-approver/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.276242596Z namespaces/openshift-cluster-machine-approver/coordination.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.276252166Z namespaces/openshift-cluster-machine-approver/coordination.k8s.io/leases/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.276281677Z namespaces/openshift-cluster-machine-approver/coordination.k8s.io/leases/cluster-machine-approver-leader.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.276353448Z namespaces/openshift-cluster-machine-approver/monitoring.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.276362128Z namespaces/openshift-cluster-machine-approver/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.276385889Z namespaces/openshift-cluster-machine-approver/monitoring.coreos.com/servicemonitors/cluster-machine-approver.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.2764488Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.27645842Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.276516431Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.276663414Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.276847698Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.276985661Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.277226616Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.277475411Z namespaces/openshift-cluster-node-tuning-operator/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.277517312Z namespaces/openshift-cluster-node-tuning-operator/openshift-cluster-node-tuning-operator.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.277635914Z namespaces/openshift-cluster-node-tuning-operator/apps.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.277673595Z namespaces/openshift-cluster-node-tuning-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.277751487Z namespaces/openshift-cluster-node-tuning-operator/apps/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.277788167Z namespaces/openshift-cluster-node-tuning-operator/apps/daemonsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.277963391Z namespaces/openshift-cluster-node-tuning-operator/apps/deployments.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.278046173Z namespaces/openshift-cluster-node-tuning-operator/apps/replicasets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.278133224Z namespaces/openshift-cluster-node-tuning-operator/apps/statefulsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.278203696Z namespaces/openshift-cluster-node-tuning-operator/apps/daemonsets/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.278239956Z namespaces/openshift-cluster-node-tuning-operator/apps/daemonsets/tuned.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.278328948Z namespaces/openshift-cluster-node-tuning-operator/autoscaling/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.278366329Z namespaces/openshift-cluster-node-tuning-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.27841132Z namespaces/openshift-cluster-node-tuning-operator/batch/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.27844643Z namespaces/openshift-cluster-node-tuning-operator/batch/cronjobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.278545023Z namespaces/openshift-cluster-node-tuning-operator/batch/jobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.278604834Z namespaces/openshift-cluster-node-tuning-operator/build.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.278642494Z namespaces/openshift-cluster-node-tuning-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.278730686Z namespaces/openshift-cluster-node-tuning-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.278776487Z namespaces/openshift-cluster-node-tuning-operator/coordination.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.278788497Z namespaces/openshift-cluster-node-tuning-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.278858749Z namespaces/openshift-cluster-node-tuning-operator/coordination.k8s.io/leases/node-tuning-operator-lock.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.27892578Z namespaces/openshift-cluster-node-tuning-operator/core/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.278989152Z namespaces/openshift-cluster-node-tuning-operator/core/configmaps.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.280092234Z namespaces/openshift-cluster-node-tuning-operator/core/endpoints.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.280237967Z namespaces/openshift-cluster-node-tuning-operator/core/events.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.28039458Z namespaces/openshift-cluster-node-tuning-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.280501962Z namespaces/openshift-cluster-node-tuning-operator/core/pods.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.280703596Z namespaces/openshift-cluster-node-tuning-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.28087382Z namespaces/openshift-cluster-node-tuning-operator/core/secrets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.281041263Z namespaces/openshift-cluster-node-tuning-operator/core/services.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.281106535Z namespaces/openshift-cluster-node-tuning-operator/core/serviceaccounts/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.281172446Z namespaces/openshift-cluster-node-tuning-operator/core/serviceaccounts/cluster-node-tuning-operator.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.281238867Z namespaces/openshift-cluster-node-tuning-operator/discovery.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.281275728Z namespaces/openshift-cluster-node-tuning-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.281331919Z namespaces/openshift-cluster-node-tuning-operator/image.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.28137494Z namespaces/openshift-cluster-node-tuning-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.281436601Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.281473852Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.281559804Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.281638695Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.281702987Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.281741277Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.281818559Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.28185569Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/prometheusrules/node-tuning-operator.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.281937541Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.281976602Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/servicemonitors/node-tuning-operator.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.282036543Z namespaces/openshift-cluster-node-tuning-operator/networking.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.282074734Z namespaces/openshift-cluster-node-tuning-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.282138195Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.282204147Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.282255728Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.282458982Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.282716097Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.28285892Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.283295459Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.283515703Z namespaces/openshift-cluster-node-tuning-operator/pods/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.283522924Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-8f9mp/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.283574905Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-8f9mp/tuned-8f9mp.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.283682647Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-8f9mp/tuned/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.283691367Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-8f9mp/tuned/tuned/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.283699177Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-8f9mp/tuned/tuned/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.283745968Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-8f9mp/tuned/tuned/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.283910991Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-8f9mp/tuned/tuned/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.283985353Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-8f9mp/tuned/tuned/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.284047504Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-x7pq9/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.284078585Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-x7pq9/tuned-x7pq9.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.284209267Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-x7pq9/tuned/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.284222308Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-x7pq9/tuned/tuned/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.284227228Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-x7pq9/tuned/tuned/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.284243078Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-x7pq9/tuned/tuned/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.28436005Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-x7pq9/tuned/tuned/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.284435652Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-x7pq9/tuned/tuned/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.284457003Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-xxpbg/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.284507204Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-xxpbg/tuned-xxpbg.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.284599105Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-xxpbg/tuned/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.284611946Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-xxpbg/tuned/tuned/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.284619086Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-xxpbg/tuned/tuned/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.284669857Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-xxpbg/tuned/tuned/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.284779889Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-xxpbg/tuned/tuned/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.2848536Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-xxpbg/tuned/tuned/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.284892881Z namespaces/openshift-cluster-node-tuning-operator/policy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.284920642Z namespaces/openshift-cluster-node-tuning-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.284983873Z namespaces/openshift-cluster-node-tuning-operator/route.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.285104685Z namespaces/openshift-cluster-node-tuning-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.285191277Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.285212688Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.285266959Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ip-10-0-135-188.ec2.internal.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.285359481Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ip-10-0-135-20.ec2.internal.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.285445443Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ip-10-0-140-52.ec2.internal.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.285504224Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/tuneds/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.285573025Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/tuneds/default.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.285705338Z namespaces/openshift-cluster-samples-operator/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.285752459Z namespaces/openshift-cluster-samples-operator/openshift-cluster-samples-operator.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.285860101Z namespaces/openshift-cluster-samples-operator/apps.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.285907402Z namespaces/openshift-cluster-samples-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.285976473Z namespaces/openshift-cluster-samples-operator/apps/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.286006404Z namespaces/openshift-cluster-samples-operator/apps/daemonsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.286105746Z namespaces/openshift-cluster-samples-operator/apps/deployments.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.28629334Z namespaces/openshift-cluster-samples-operator/apps/replicasets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.286391662Z namespaces/openshift-cluster-samples-operator/apps/statefulsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.286455593Z namespaces/openshift-cluster-samples-operator/autoscaling/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.286503114Z namespaces/openshift-cluster-samples-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.286551715Z namespaces/openshift-cluster-samples-operator/batch/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.286617796Z namespaces/openshift-cluster-samples-operator/batch/cronjobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.286730799Z namespaces/openshift-cluster-samples-operator/batch/jobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.28679002Z namespaces/openshift-cluster-samples-operator/build.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.2868262Z namespaces/openshift-cluster-samples-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.286909232Z namespaces/openshift-cluster-samples-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.286986624Z namespaces/openshift-cluster-samples-operator/core/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.287033885Z namespaces/openshift-cluster-samples-operator/core/configmaps.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.287246989Z namespaces/openshift-cluster-samples-operator/core/endpoints.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.287339611Z namespaces/openshift-cluster-samples-operator/core/events.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.287514874Z namespaces/openshift-cluster-samples-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.287639037Z namespaces/openshift-cluster-samples-operator/core/pods.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.28776287Z namespaces/openshift-cluster-samples-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.287907473Z namespaces/openshift-cluster-samples-operator/core/secrets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.288108897Z namespaces/openshift-cluster-samples-operator/core/services.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.288582866Z namespaces/openshift-cluster-samples-operator/discovery.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.288633667Z namespaces/openshift-cluster-samples-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.288708849Z namespaces/openshift-cluster-samples-operator/image.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.28875566Z namespaces/openshift-cluster-samples-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.288826141Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.288865152Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.288948494Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.289065936Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.289089026Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.289154788Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.289226959Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.28927279Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/prometheusrules/samples-operator-alerts.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.289341342Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.289390403Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/servicemonitors/cluster-samples-operator.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.289423963Z namespaces/openshift-cluster-samples-operator/networking.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.289476344Z namespaces/openshift-cluster-samples-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.289508445Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.289513895Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.289565686Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.289711269Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.289893763Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.290034886Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.290285051Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.290470654Z namespaces/openshift-cluster-samples-operator/pods/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.290483175Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-rp5rp/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.290509515Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-rp5rp/cluster-samples-operator-6748489656-rp5rp.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.290625658Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-rp5rp/cluster-samples-operator-watch/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.290635108Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-rp5rp/cluster-samples-operator-watch/cluster-samples-operator-watch/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.290639428Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-rp5rp/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.290685479Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-rp5rp/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.290775701Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-rp5rp/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.290852892Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-rp5rp/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.290912894Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-rp5rp/cluster-samples-operator/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.290924764Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-rp5rp/cluster-samples-operator/cluster-samples-operator/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.290932894Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-rp5rp/cluster-samples-operator/cluster-samples-operator/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.290965544Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-rp5rp/cluster-samples-operator/cluster-samples-operator/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.29172328Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-rp5rp/cluster-samples-operator/cluster-samples-operator/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.291802701Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-rp5rp/cluster-samples-operator/cluster-samples-operator/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.291843732Z namespaces/openshift-cluster-samples-operator/policy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.291877803Z namespaces/openshift-cluster-samples-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.291942334Z namespaces/openshift-cluster-samples-operator/route.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.291979975Z namespaces/openshift-cluster-samples-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.292075327Z namespaces/openshift-cluster-storage-operator/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.292144528Z namespaces/openshift-cluster-storage-operator/openshift-cluster-storage-operator.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.29221992Z namespaces/openshift-cluster-storage-operator/apps.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.292263341Z namespaces/openshift-cluster-storage-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.292334872Z namespaces/openshift-cluster-storage-operator/apps/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.292385633Z namespaces/openshift-cluster-storage-operator/apps/daemonsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.292498636Z namespaces/openshift-cluster-storage-operator/apps/deployments.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.292607188Z namespaces/openshift-cluster-storage-operator/apps/replicasets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.29272022Z namespaces/openshift-cluster-storage-operator/apps/statefulsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.292774591Z namespaces/openshift-cluster-storage-operator/autoscaling/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.292820152Z namespaces/openshift-cluster-storage-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.292880453Z namespaces/openshift-cluster-storage-operator/batch/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.292936975Z namespaces/openshift-cluster-storage-operator/batch/cronjobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.293021256Z namespaces/openshift-cluster-storage-operator/batch/jobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.293077857Z namespaces/openshift-cluster-storage-operator/build.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.293136819Z namespaces/openshift-cluster-storage-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.293261401Z namespaces/openshift-cluster-storage-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.293307902Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.293325742Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.293372143Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/leases/data-source-validator-leader.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.293488386Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/leases/snapshot-controller-leader.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.293569387Z namespaces/openshift-cluster-storage-operator/core/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.293614528Z namespaces/openshift-cluster-storage-operator/core/configmaps.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.293735311Z namespaces/openshift-cluster-storage-operator/core/endpoints.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.293816953Z namespaces/openshift-cluster-storage-operator/core/events.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.29418305Z namespaces/openshift-cluster-storage-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.294268612Z namespaces/openshift-cluster-storage-operator/core/pods.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.294383854Z namespaces/openshift-cluster-storage-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.294541407Z namespaces/openshift-cluster-storage-operator/core/secrets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.29469859Z namespaces/openshift-cluster-storage-operator/core/services.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.294761601Z namespaces/openshift-cluster-storage-operator/core/serviceaccounts/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.294810793Z namespaces/openshift-cluster-storage-operator/core/serviceaccounts/csi-snapshot-controller-operator.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.294899334Z namespaces/openshift-cluster-storage-operator/discovery.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.294922725Z namespaces/openshift-cluster-storage-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.295009366Z namespaces/openshift-cluster-storage-operator/image.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.295027957Z namespaces/openshift-cluster-storage-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.295117639Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.29516779Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.295255771Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.295342553Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.295412915Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.295455206Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.295524267Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.295566618Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/prometheusrules/prometheus.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.29566284Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.295705341Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/servicemonitors/cluster-storage-operator.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.295763162Z namespaces/openshift-cluster-storage-operator/networking.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.295794433Z namespaces/openshift-cluster-storage-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.295881464Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.295892755Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.295938205Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.296093139Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.296289123Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.296424615Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.296694651Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.296878815Z namespaces/openshift-cluster-storage-operator/pods/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.296885795Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-fsv8l/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.296920175Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-fsv8l/volume-data-source-validator-549c6d54cf-fsv8l.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.297000997Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-fsv8l/volume-data-source-validator/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.297008247Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-fsv8l/volume-data-source-validator/volume-data-source-validator/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.297012377Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-fsv8l/volume-data-source-validator/volume-data-source-validator/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.297040158Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-fsv8l/volume-data-source-validator/volume-data-source-validator/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.297207111Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-fsv8l/volume-data-source-validator/volume-data-source-validator/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.297403405Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-fsv8l/volume-data-source-validator/volume-data-source-validator/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.297443426Z namespaces/openshift-cluster-storage-operator/policy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.297560058Z namespaces/openshift-cluster-storage-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.297608689Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.297615379Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.29766038Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/rolebindings/csi-snapshot-controller-operator-role.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.297728982Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.297769213Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/roles/csi-snapshot-controller-operator-role.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.297836804Z namespaces/openshift-cluster-storage-operator/route.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.297877065Z namespaces/openshift-cluster-storage-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.297978657Z namespaces/openshift-cluster-version/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.298066509Z namespaces/openshift-cluster-version/openshift-cluster-version.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.298194901Z namespaces/openshift-cluster-version/apps.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.298263863Z namespaces/openshift-cluster-version/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.298327844Z namespaces/openshift-cluster-version/apps/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.298371125Z namespaces/openshift-cluster-version/apps/daemonsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.298459887Z namespaces/openshift-cluster-version/apps/deployments.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.298535158Z namespaces/openshift-cluster-version/apps/replicasets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.29862901Z namespaces/openshift-cluster-version/apps/statefulsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.298675691Z namespaces/openshift-cluster-version/autoscaling/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.298728642Z namespaces/openshift-cluster-version/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.298789143Z namespaces/openshift-cluster-version/batch/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.298835954Z namespaces/openshift-cluster-version/batch/cronjobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.298925266Z namespaces/openshift-cluster-version/batch/jobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.298985797Z namespaces/openshift-cluster-version/build.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.299024168Z namespaces/openshift-cluster-version/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.29910872Z namespaces/openshift-cluster-version/build.openshift.io/builds.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.299268243Z namespaces/openshift-cluster-version/coordination.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.299279603Z namespaces/openshift-cluster-version/coordination.k8s.io/leases/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.299321744Z namespaces/openshift-cluster-version/coordination.k8s.io/leases/version.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.299403806Z namespaces/openshift-cluster-version/core/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.299446347Z namespaces/openshift-cluster-version/core/configmaps.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.299546499Z namespaces/openshift-cluster-version/core/endpoints.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.29963396Z namespaces/openshift-cluster-version/core/events.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.299732542Z namespaces/openshift-cluster-version/core/persistentvolumeclaims.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.299812794Z namespaces/openshift-cluster-version/core/pods.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.299892256Z namespaces/openshift-cluster-version/core/replicationcontrollers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.300036269Z namespaces/openshift-cluster-version/core/secrets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.300252683Z namespaces/openshift-cluster-version/core/services.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.300330945Z namespaces/openshift-cluster-version/discovery.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.300369915Z namespaces/openshift-cluster-version/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.300431957Z namespaces/openshift-cluster-version/image.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.300469097Z namespaces/openshift-cluster-version/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.300517728Z namespaces/openshift-cluster-version/k8s.ovn.org/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.300551809Z namespaces/openshift-cluster-version/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.300635351Z namespaces/openshift-cluster-version/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.300716612Z namespaces/openshift-cluster-version/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.300770223Z namespaces/openshift-cluster-version/monitoring.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.300807254Z namespaces/openshift-cluster-version/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.300862535Z namespaces/openshift-cluster-version/networking.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.300899486Z namespaces/openshift-cluster-version/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.300955977Z namespaces/openshift-cluster-version/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.300965598Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.301018259Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.301198552Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.301368666Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.301504008Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.301738623Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.301930617Z namespaces/openshift-cluster-version/policy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.301950907Z namespaces/openshift-cluster-version/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.302022599Z namespaces/openshift-cluster-version/route.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.302043929Z namespaces/openshift-cluster-version/route.openshift.io/routes.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.302183462Z namespaces/openshift-config-managed/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.302244884Z namespaces/openshift-config-managed/openshift-config-managed.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.302308085Z namespaces/openshift-config-managed/apps.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.302371936Z namespaces/openshift-config-managed/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.302428677Z namespaces/openshift-config-managed/apps/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.302482578Z namespaces/openshift-config-managed/apps/daemonsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.30258701Z namespaces/openshift-config-managed/apps/deployments.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.302655932Z namespaces/openshift-config-managed/apps/replicasets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.302742683Z namespaces/openshift-config-managed/apps/statefulsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.302778604Z namespaces/openshift-config-managed/autoscaling/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.302829015Z namespaces/openshift-config-managed/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.302876966Z namespaces/openshift-config-managed/batch/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.302924747Z namespaces/openshift-config-managed/batch/cronjobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.303003499Z namespaces/openshift-config-managed/batch/jobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.30306831Z namespaces/openshift-config-managed/build.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.303145292Z namespaces/openshift-config-managed/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.303232933Z namespaces/openshift-config-managed/build.openshift.io/builds.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.303299075Z namespaces/openshift-config-managed/core/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.303346056Z namespaces/openshift-config-managed/core/configmaps.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.306351527Z namespaces/openshift-config-managed/core/endpoints.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.306428958Z namespaces/openshift-config-managed/core/events.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.30652325Z namespaces/openshift-config-managed/core/persistentvolumeclaims.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.306623082Z namespaces/openshift-config-managed/core/pods.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.306733764Z namespaces/openshift-config-managed/core/replicationcontrollers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.306896728Z namespaces/openshift-config-managed/core/secrets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.30701713Z namespaces/openshift-config-managed/core/services.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.307078922Z namespaces/openshift-config-managed/core/configmaps/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.307147193Z namespaces/openshift-config-managed/core/configmaps/console-public.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.307248925Z namespaces/openshift-config-managed/core/configmaps/openshift-network-features.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.307319996Z namespaces/openshift-config-managed/discovery.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.307354997Z namespaces/openshift-config-managed/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.307392068Z namespaces/openshift-config-managed/image.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.307450719Z namespaces/openshift-config-managed/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.30749175Z namespaces/openshift-config-managed/k8s.ovn.org/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.307569641Z namespaces/openshift-config-managed/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.307681264Z namespaces/openshift-config-managed/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.307794666Z namespaces/openshift-config-managed/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.307840607Z namespaces/openshift-config-managed/monitoring.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.307882108Z namespaces/openshift-config-managed/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.307941209Z namespaces/openshift-config-managed/networking.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.3079899Z namespaces/openshift-config-managed/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.308049061Z namespaces/openshift-config-managed/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.308059141Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.308113883Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.308290536Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.3085056Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.308649843Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.308859748Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.309149444Z namespaces/openshift-config-managed/policy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.309316907Z namespaces/openshift-config-managed/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.309356008Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.309375578Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.309429589Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/rolebindings/openshift-network-public-role-binding.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.30947041Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.309536831Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/roles/openshift-network-public-role.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.309588002Z namespaces/openshift-config-managed/route.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.309630733Z namespaces/openshift-config-managed/route.openshift.io/routes.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.309667154Z namespaces/openshift-config-operator/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.309684154Z namespaces/openshift-config-operator/monitoring.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.309689644Z namespaces/openshift-config-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.309748696Z namespaces/openshift-config-operator/monitoring.coreos.com/servicemonitors/config-operator.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.309791686Z namespaces/openshift-config-operator/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.309804447Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.309872948Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.310021421Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.310233975Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.310383919Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.310698185Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.311159324Z namespaces/openshift-config/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.311234156Z namespaces/openshift-config/openshift-config.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.311306247Z namespaces/openshift-config/apps.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.311325158Z namespaces/openshift-config/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.31143375Z namespaces/openshift-config/apps/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.311495011Z namespaces/openshift-config/apps/daemonsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.311612094Z namespaces/openshift-config/apps/deployments.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.311804758Z namespaces/openshift-config/apps/replicasets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.311892739Z namespaces/openshift-config/apps/statefulsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.3119481Z namespaces/openshift-config/autoscaling/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.311991381Z namespaces/openshift-config/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.312056252Z namespaces/openshift-config/batch/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.312113064Z namespaces/openshift-config/batch/cronjobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.312219546Z namespaces/openshift-config/batch/jobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.312259787Z namespaces/openshift-config/build.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.312292497Z namespaces/openshift-config/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.312382919Z namespaces/openshift-config/build.openshift.io/builds.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.31242829Z namespaces/openshift-config/core/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.312465621Z namespaces/openshift-config/core/configmaps.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.312582283Z namespaces/openshift-config/core/endpoints.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.312657345Z namespaces/openshift-config/core/events.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.312742666Z namespaces/openshift-config/core/persistentvolumeclaims.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.312818308Z namespaces/openshift-config/core/pods.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.312895609Z namespaces/openshift-config/core/replicationcontrollers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.313089384Z namespaces/openshift-config/core/secrets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.313273907Z namespaces/openshift-config/core/services.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.313332798Z namespaces/openshift-config/core/secrets/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.313368229Z namespaces/openshift-config/core/secrets/pull-secret.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.31342898Z namespaces/openshift-config/discovery.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.313477001Z namespaces/openshift-config/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.313534373Z namespaces/openshift-config/image.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.313571093Z namespaces/openshift-config/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.313639105Z namespaces/openshift-config/k8s.ovn.org/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.313687385Z namespaces/openshift-config/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.313773737Z namespaces/openshift-config/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.313850449Z namespaces/openshift-config/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.3139051Z namespaces/openshift-config/monitoring.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.313974922Z namespaces/openshift-config/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.314039463Z namespaces/openshift-config/networking.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.314081673Z namespaces/openshift-config/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.314167605Z namespaces/openshift-config/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.314184616Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.314218196Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.314560513Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.314754977Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.3148855Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.315073534Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.315273438Z namespaces/openshift-config/policy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.315324129Z namespaces/openshift-config/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.31537401Z namespaces/openshift-config/route.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.315408811Z namespaces/openshift-config/route.openshift.io/routes.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.315496112Z namespaces/openshift-console-operator/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.315544713Z namespaces/openshift-console-operator/openshift-console-operator.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.315616155Z namespaces/openshift-console-operator/apps.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.315643115Z namespaces/openshift-console-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.315726017Z namespaces/openshift-console-operator/apps/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.315762668Z namespaces/openshift-console-operator/apps/daemonsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.31586151Z namespaces/openshift-console-operator/apps/deployments.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.315984582Z namespaces/openshift-console-operator/apps/replicasets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.316155486Z namespaces/openshift-console-operator/apps/statefulsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.316324759Z namespaces/openshift-console-operator/autoscaling/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.3163503Z namespaces/openshift-console-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.316430661Z namespaces/openshift-console-operator/batch/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.316760718Z namespaces/openshift-console-operator/batch/cronjobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.316786468Z namespaces/openshift-console-operator/batch/jobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.316793809Z namespaces/openshift-console-operator/build.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.316799679Z namespaces/openshift-console-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.316899871Z namespaces/openshift-console-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.316922361Z namespaces/openshift-console-operator/coordination.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.316929822Z namespaces/openshift-console-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.317000673Z namespaces/openshift-console-operator/coordination.k8s.io/leases/console-operator-lock.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.317088445Z namespaces/openshift-console-operator/core/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.317153106Z namespaces/openshift-console-operator/core/configmaps.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.317777459Z namespaces/openshift-console-operator/core/endpoints.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.317880621Z namespaces/openshift-console-operator/core/events.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.318210828Z namespaces/openshift-console-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.31834066Z namespaces/openshift-console-operator/core/pods.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.318502343Z namespaces/openshift-console-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.318703857Z namespaces/openshift-console-operator/core/secrets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.318939342Z namespaces/openshift-console-operator/core/services.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.319016914Z namespaces/openshift-console-operator/discovery.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.319062695Z namespaces/openshift-console-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.319141096Z namespaces/openshift-console-operator/image.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.319184457Z namespaces/openshift-console-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.319260969Z namespaces/openshift-console-operator/k8s.ovn.org/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.319303499Z namespaces/openshift-console-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.319418242Z namespaces/openshift-console-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.319488883Z namespaces/openshift-console-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.319564575Z namespaces/openshift-console-operator/monitoring.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.319607906Z namespaces/openshift-console-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.319694498Z namespaces/openshift-console-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.319726638Z namespaces/openshift-console-operator/monitoring.coreos.com/prometheusrules/cluster-monitoring-prometheus-rules.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.31981501Z namespaces/openshift-console-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.319855511Z namespaces/openshift-console-operator/monitoring.coreos.com/servicemonitors/console-operator.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.319921932Z namespaces/openshift-console-operator/networking.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.319973193Z namespaces/openshift-console-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.320019824Z namespaces/openshift-console-operator/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.320033805Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.320092166Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.32028695Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.320478094Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.320637767Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.320899052Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.321245159Z namespaces/openshift-console-operator/pods/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.321259169Z namespaces/openshift-console-operator/pods/console-operator-584dcb6b54-c679z/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.32131208Z namespaces/openshift-console-operator/pods/console-operator-584dcb6b54-c679z/console-operator-584dcb6b54-c679z.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.321414232Z namespaces/openshift-console-operator/pods/console-operator-584dcb6b54-c679z/console-operator/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.321427793Z namespaces/openshift-console-operator/pods/console-operator-584dcb6b54-c679z/console-operator/console-operator/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.321434213Z namespaces/openshift-console-operator/pods/console-operator-584dcb6b54-c679z/console-operator/console-operator/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.321475484Z namespaces/openshift-console-operator/pods/console-operator-584dcb6b54-c679z/console-operator/console-operator/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.322024475Z namespaces/openshift-console-operator/pods/console-operator-584dcb6b54-c679z/console-operator/console-operator/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.322084466Z namespaces/openshift-console-operator/policy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.322145007Z namespaces/openshift-console-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.322214589Z namespaces/openshift-console-operator/route.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.32227706Z namespaces/openshift-console-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.322335391Z namespaces/openshift-console-user-settings/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.322348351Z namespaces/openshift-console-user-settings/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.322356332Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.322456234Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.322678048Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.322878012Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.32328382Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.323526905Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.32376783Z namespaces/openshift-console/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.323852312Z namespaces/openshift-console/openshift-console.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.324340372Z namespaces/openshift-console/apps.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.324422454Z namespaces/openshift-console/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.324488145Z namespaces/openshift-console/apps/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.324549646Z namespaces/openshift-console/apps/daemonsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.324623347Z namespaces/openshift-console/apps/deployments.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.324796241Z namespaces/openshift-console/apps/replicasets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.325074087Z namespaces/openshift-console/apps/statefulsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.325114797Z namespaces/openshift-console/autoscaling/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.325207999Z namespaces/openshift-console/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.325266881Z namespaces/openshift-console/batch/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.325319412Z namespaces/openshift-console/batch/cronjobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.325401563Z namespaces/openshift-console/batch/jobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.325456835Z namespaces/openshift-console/build.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.325506536Z namespaces/openshift-console/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.325579507Z namespaces/openshift-console/build.openshift.io/builds.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.325631998Z namespaces/openshift-console/core/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.325670839Z namespaces/openshift-console/core/configmaps.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.326288291Z namespaces/openshift-console/core/endpoints.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.326373723Z namespaces/openshift-console/core/events.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.326652919Z namespaces/openshift-console/core/persistentvolumeclaims.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.3267294Z namespaces/openshift-console/core/pods.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.326922704Z namespaces/openshift-console/core/replicationcontrollers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.327057587Z namespaces/openshift-console/core/secrets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.327295142Z namespaces/openshift-console/core/services.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.327354103Z namespaces/openshift-console/discovery.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.327400144Z namespaces/openshift-console/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.327481236Z namespaces/openshift-console/image.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.327526357Z namespaces/openshift-console/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.327613128Z namespaces/openshift-console/k8s.ovn.org/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.327665519Z namespaces/openshift-console/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.327756761Z namespaces/openshift-console/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.327843153Z namespaces/openshift-console/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.327895974Z namespaces/openshift-console/monitoring.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.327944255Z namespaces/openshift-console/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.328005936Z namespaces/openshift-console/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.328017196Z namespaces/openshift-console/monitoring.coreos.com/servicemonitors/console.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.328087978Z namespaces/openshift-console/networking.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.328140049Z namespaces/openshift-console/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.3282155Z namespaces/openshift-console/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.328229971Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.328277602Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.328428375Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.328613819Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.328743541Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.328963856Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.32918816Z namespaces/openshift-console/pods/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.329205471Z namespaces/openshift-console/pods/console-c4fff4cb5-l85lf/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.329257101Z namespaces/openshift-console/pods/console-c4fff4cb5-l85lf/console-c4fff4cb5-l85lf.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.329355814Z namespaces/openshift-console/pods/console-c4fff4cb5-l85lf/console/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.329364994Z namespaces/openshift-console/pods/console-c4fff4cb5-l85lf/console/console/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.329370314Z namespaces/openshift-console/pods/console-c4fff4cb5-l85lf/console/console/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.329401055Z namespaces/openshift-console/pods/console-c4fff4cb5-l85lf/console/console/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.329504497Z namespaces/openshift-console/pods/console-c4fff4cb5-l85lf/console/console/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.329568308Z namespaces/openshift-console/pods/console-c4fff4cb5-l85lf/console/console/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.329609509Z namespaces/openshift-console/pods/downloads-7c97dc4555-2vcxq/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.329637389Z namespaces/openshift-console/pods/downloads-7c97dc4555-2vcxq/downloads-7c97dc4555-2vcxq.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.329741751Z namespaces/openshift-console/pods/downloads-7c97dc4555-2vcxq/download-server/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.329750742Z namespaces/openshift-console/pods/downloads-7c97dc4555-2vcxq/download-server/download-server/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.329759502Z namespaces/openshift-console/pods/downloads-7c97dc4555-2vcxq/download-server/download-server/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.329768492Z namespaces/openshift-console/pods/downloads-7c97dc4555-2vcxq/download-server/download-server/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.3301464Z namespaces/openshift-console/pods/downloads-7c97dc4555-2vcxq/download-server/download-server/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.330234101Z namespaces/openshift-console/pods/downloads-7c97dc4555-2vcxq/download-server/download-server/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.330274392Z namespaces/openshift-console/policy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.330308943Z namespaces/openshift-console/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.330393105Z namespaces/openshift-console/route.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.330416005Z namespaces/openshift-console/route.openshift.io/routes.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.330520277Z namespaces/openshift-controller-manager-operator/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.330560788Z namespaces/openshift-controller-manager-operator/openshift-controller-manager-operator.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.33063696Z namespaces/openshift-controller-manager-operator/apps.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.33067563Z namespaces/openshift-controller-manager-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.330725751Z namespaces/openshift-controller-manager-operator/apps/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.330760872Z namespaces/openshift-controller-manager-operator/apps/daemonsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.330848204Z namespaces/openshift-controller-manager-operator/apps/deployments.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.330924846Z namespaces/openshift-controller-manager-operator/apps/replicasets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.331022308Z namespaces/openshift-controller-manager-operator/apps/statefulsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.331042478Z namespaces/openshift-controller-manager-operator/autoscaling/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.331088829Z namespaces/openshift-controller-manager-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.331186781Z namespaces/openshift-controller-manager-operator/batch/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.331253632Z namespaces/openshift-controller-manager-operator/batch/cronjobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.331324174Z namespaces/openshift-controller-manager-operator/batch/jobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.331375704Z namespaces/openshift-controller-manager-operator/build.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.331425596Z namespaces/openshift-controller-manager-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.331501477Z namespaces/openshift-controller-manager-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.331569548Z namespaces/openshift-controller-manager-operator/core/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.331607689Z namespaces/openshift-controller-manager-operator/core/configmaps.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.331725902Z namespaces/openshift-controller-manager-operator/core/endpoints.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.331797513Z namespaces/openshift-controller-manager-operator/core/events.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.331882025Z namespaces/openshift-controller-manager-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.331960987Z namespaces/openshift-controller-manager-operator/core/pods.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.332044178Z namespaces/openshift-controller-manager-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.332319954Z namespaces/openshift-controller-manager-operator/core/secrets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.332686601Z namespaces/openshift-controller-manager-operator/core/services.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.332728582Z namespaces/openshift-controller-manager-operator/discovery.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.332768283Z namespaces/openshift-controller-manager-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.332833074Z namespaces/openshift-controller-manager-operator/image.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.332887925Z namespaces/openshift-controller-manager-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.332956407Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.333002538Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.333096719Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.333203422Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.333224572Z namespaces/openshift-controller-manager-operator/monitoring.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.333281613Z namespaces/openshift-controller-manager-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.333322944Z namespaces/openshift-controller-manager-operator/networking.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.333345424Z namespaces/openshift-controller-manager-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.333417536Z namespaces/openshift-controller-manager-operator/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.333430446Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.333439526Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.3335922Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.333816544Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.333988728Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.334228393Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.334471417Z namespaces/openshift-controller-manager-operator/policy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.334506938Z namespaces/openshift-controller-manager-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.33457728Z namespaces/openshift-controller-manager-operator/route.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.33461302Z namespaces/openshift-controller-manager-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.334717502Z namespaces/openshift-controller-manager/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.334762023Z namespaces/openshift-controller-manager/openshift-controller-manager.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.334821574Z namespaces/openshift-controller-manager/apps.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.334853705Z namespaces/openshift-controller-manager/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.334915016Z namespaces/openshift-controller-manager/apps/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.334969837Z namespaces/openshift-controller-manager/apps/daemonsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.335070059Z namespaces/openshift-controller-manager/apps/deployments.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.335189092Z namespaces/openshift-controller-manager/apps/replicasets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.335269583Z namespaces/openshift-controller-manager/apps/statefulsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.335381506Z namespaces/openshift-controller-manager/autoscaling/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.335432477Z namespaces/openshift-controller-manager/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.335489108Z namespaces/openshift-controller-manager/batch/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.335536329Z namespaces/openshift-controller-manager/batch/cronjobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.335620671Z namespaces/openshift-controller-manager/batch/jobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.335668742Z namespaces/openshift-controller-manager/build.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.335709792Z namespaces/openshift-controller-manager/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.335789914Z namespaces/openshift-controller-manager/build.openshift.io/builds.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.335842405Z namespaces/openshift-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.335849485Z namespaces/openshift-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.335899236Z namespaces/openshift-controller-manager/coordination.k8s.io/leases/openshift-master-controllers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.335971738Z namespaces/openshift-controller-manager/core/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.336014839Z namespaces/openshift-controller-manager/core/configmaps.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.336190832Z namespaces/openshift-controller-manager/core/endpoints.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.336290124Z namespaces/openshift-controller-manager/core/events.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.336371626Z namespaces/openshift-controller-manager/core/persistentvolumeclaims.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.336447678Z namespaces/openshift-controller-manager/core/pods.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.336536669Z namespaces/openshift-controller-manager/core/replicationcontrollers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.336676712Z namespaces/openshift-controller-manager/core/secrets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.336855746Z namespaces/openshift-controller-manager/core/services.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.337028219Z namespaces/openshift-controller-manager/discovery.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.33708486Z namespaces/openshift-controller-manager/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.337140452Z namespaces/openshift-controller-manager/image.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.337200683Z namespaces/openshift-controller-manager/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.337255394Z namespaces/openshift-controller-manager/k8s.ovn.org/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.337327925Z namespaces/openshift-controller-manager/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.337418047Z namespaces/openshift-controller-manager/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.337498109Z namespaces/openshift-controller-manager/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.33755063Z namespaces/openshift-controller-manager/monitoring.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.337586651Z namespaces/openshift-controller-manager/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.337635732Z namespaces/openshift-controller-manager/networking.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.337673422Z namespaces/openshift-controller-manager/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.337728533Z namespaces/openshift-controller-manager/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.337736104Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.337768504Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.337928678Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.338093701Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.338258844Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.338461839Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.338657292Z namespaces/openshift-controller-manager/policy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.338695773Z namespaces/openshift-controller-manager/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.338744244Z namespaces/openshift-controller-manager/route.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.338794825Z namespaces/openshift-controller-manager/route.openshift.io/routes.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.338878147Z namespaces/openshift-dns-operator/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.338925798Z namespaces/openshift-dns-operator/openshift-dns-operator.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.338979169Z namespaces/openshift-dns-operator/apps.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.33901931Z namespaces/openshift-dns-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.339089721Z namespaces/openshift-dns-operator/apps/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.339149352Z namespaces/openshift-dns-operator/apps/daemonsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.339252395Z namespaces/openshift-dns-operator/apps/deployments.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.339344906Z namespaces/openshift-dns-operator/apps/replicasets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.339416248Z namespaces/openshift-dns-operator/apps/statefulsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.33954853Z namespaces/openshift-dns-operator/autoscaling/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.339585161Z namespaces/openshift-dns-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.339642372Z namespaces/openshift-dns-operator/batch/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.339680003Z namespaces/openshift-dns-operator/batch/cronjobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.339778745Z namespaces/openshift-dns-operator/batch/jobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.339856467Z namespaces/openshift-dns-operator/build.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.339893607Z namespaces/openshift-dns-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.339980169Z namespaces/openshift-dns-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.340044771Z namespaces/openshift-dns-operator/core/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.340095581Z namespaces/openshift-dns-operator/core/configmaps.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.340235404Z namespaces/openshift-dns-operator/core/endpoints.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.340325446Z namespaces/openshift-dns-operator/core/events.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.340411518Z namespaces/openshift-dns-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.3404903Z namespaces/openshift-dns-operator/core/pods.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.340570181Z namespaces/openshift-dns-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.340699304Z namespaces/openshift-dns-operator/core/secrets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.340879168Z namespaces/openshift-dns-operator/core/services.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.340952729Z namespaces/openshift-dns-operator/discovery.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.341035231Z namespaces/openshift-dns-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.341084852Z namespaces/openshift-dns-operator/image.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.341137023Z namespaces/openshift-dns-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.341264195Z namespaces/openshift-dns-operator/k8s.ovn.org/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.341301446Z namespaces/openshift-dns-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.341386038Z namespaces/openshift-dns-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.341457599Z namespaces/openshift-dns-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.34152001Z namespaces/openshift-dns-operator/monitoring.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.341582162Z namespaces/openshift-dns-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.341620172Z namespaces/openshift-dns-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.341681414Z namespaces/openshift-dns-operator/monitoring.coreos.com/prometheusrules/dns.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.341745195Z namespaces/openshift-dns-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.341772926Z namespaces/openshift-dns-operator/monitoring.coreos.com/servicemonitors/dns-operator.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.341831557Z namespaces/openshift-dns-operator/networking.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.341859427Z namespaces/openshift-dns-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.341913809Z namespaces/openshift-dns-operator/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.341921458Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.341957299Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.342107322Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.342305636Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.34245698Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.342704794Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.342903438Z namespaces/openshift-dns-operator/policy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.342940819Z namespaces/openshift-dns-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.343007911Z namespaces/openshift-dns-operator/route.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.343053662Z namespaces/openshift-dns-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.343169624Z namespaces/openshift-dns/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.343212345Z namespaces/openshift-dns/openshift-dns.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.343282506Z namespaces/openshift-dns/apps.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.343318067Z namespaces/openshift-dns/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.344023841Z namespaces/openshift-dns/apps/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.344103333Z namespaces/openshift-dns/apps/daemonsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.344305477Z namespaces/openshift-dns/apps/deployments.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.344366078Z namespaces/openshift-dns/apps/replicasets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.3444489Z namespaces/openshift-dns/apps/statefulsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.344505661Z namespaces/openshift-dns/autoscaling/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.344524181Z namespaces/openshift-dns/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.344591803Z namespaces/openshift-dns/batch/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.344635614Z namespaces/openshift-dns/batch/cronjobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.344731655Z namespaces/openshift-dns/batch/jobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.344789147Z namespaces/openshift-dns/build.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.344835388Z namespaces/openshift-dns/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.34494894Z namespaces/openshift-dns/build.openshift.io/builds.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.345019261Z namespaces/openshift-dns/core/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.345074013Z namespaces/openshift-dns/core/configmaps.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.345211025Z namespaces/openshift-dns/core/endpoints.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.345294627Z namespaces/openshift-dns/core/events.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.345511482Z namespaces/openshift-dns/core/persistentvolumeclaims.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.345587113Z namespaces/openshift-dns/core/pods.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.345856938Z namespaces/openshift-dns/core/replicationcontrollers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.345983321Z namespaces/openshift-dns/core/secrets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.346202005Z namespaces/openshift-dns/core/services.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.346268637Z namespaces/openshift-dns/discovery.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.346315458Z namespaces/openshift-dns/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.346572133Z namespaces/openshift-dns/image.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.346603394Z namespaces/openshift-dns/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.346673115Z namespaces/openshift-dns/k8s.ovn.org/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.346703706Z namespaces/openshift-dns/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.346797448Z namespaces/openshift-dns/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.346879689Z namespaces/openshift-dns/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.3469426Z namespaces/openshift-dns/monitoring.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.346986421Z namespaces/openshift-dns/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.347048823Z namespaces/openshift-dns/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.347097094Z namespaces/openshift-dns/monitoring.coreos.com/servicemonitors/dns-default.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.347152995Z namespaces/openshift-dns/networking.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.347201606Z namespaces/openshift-dns/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.347246687Z namespaces/openshift-dns/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.347256057Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.347278997Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.347444501Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.347611284Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.347752797Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.347970511Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.348240907Z namespaces/openshift-dns/pods/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.348259197Z namespaces/openshift-dns/pods/dns-default-2nw7c/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.348301448Z namespaces/openshift-dns/pods/dns-default-2nw7c/dns-default-2nw7c.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.34839141Z namespaces/openshift-dns/pods/dns-default-2nw7c/dns/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.34839986Z namespaces/openshift-dns/pods/dns-default-2nw7c/dns/dns/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.34841044Z namespaces/openshift-dns/pods/dns-default-2nw7c/dns/dns/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.348451671Z namespaces/openshift-dns/pods/dns-default-2nw7c/dns/dns/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.348558393Z namespaces/openshift-dns/pods/dns-default-2nw7c/dns/dns/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.348655075Z namespaces/openshift-dns/pods/dns-default-2nw7c/dns/dns/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.348691016Z namespaces/openshift-dns/pods/dns-default-2nw7c/kube-rbac-proxy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.348700506Z namespaces/openshift-dns/pods/dns-default-2nw7c/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.348712696Z namespaces/openshift-dns/pods/dns-default-2nw7c/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.348758997Z namespaces/openshift-dns/pods/dns-default-2nw7c/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.34886945Z namespaces/openshift-dns/pods/dns-default-2nw7c/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.348946451Z namespaces/openshift-dns/pods/dns-default-2nw7c/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.348979612Z namespaces/openshift-dns/pods/dns-default-cqcfv/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.349050773Z namespaces/openshift-dns/pods/dns-default-cqcfv/dns-default-cqcfv.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.349157316Z namespaces/openshift-dns/pods/dns-default-cqcfv/dns/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.349239637Z namespaces/openshift-dns/pods/dns-default-cqcfv/dns/dns/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.349256198Z namespaces/openshift-dns/pods/dns-default-cqcfv/dns/dns/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.349303538Z namespaces/openshift-dns/pods/dns-default-cqcfv/dns/dns/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.349414761Z namespaces/openshift-dns/pods/dns-default-cqcfv/dns/dns/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.349482262Z namespaces/openshift-dns/pods/dns-default-cqcfv/dns/dns/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.349539853Z namespaces/openshift-dns/pods/dns-default-cqcfv/kube-rbac-proxy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.349553633Z namespaces/openshift-dns/pods/dns-default-cqcfv/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.349564774Z namespaces/openshift-dns/pods/dns-default-cqcfv/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.349590844Z namespaces/openshift-dns/pods/dns-default-cqcfv/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.349691926Z namespaces/openshift-dns/pods/dns-default-cqcfv/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.349785158Z namespaces/openshift-dns/pods/dns-default-cqcfv/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.349841509Z namespaces/openshift-dns/pods/dns-default-f96fd/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.34988614Z namespaces/openshift-dns/pods/dns-default-f96fd/dns-default-f96fd.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.349957162Z namespaces/openshift-dns/pods/dns-default-f96fd/dns/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.349966302Z namespaces/openshift-dns/pods/dns-default-f96fd/dns/dns/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.349979882Z namespaces/openshift-dns/pods/dns-default-f96fd/dns/dns/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.350027633Z namespaces/openshift-dns/pods/dns-default-f96fd/dns/dns/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.350140345Z namespaces/openshift-dns/pods/dns-default-f96fd/dns/dns/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.350219767Z namespaces/openshift-dns/pods/dns-default-f96fd/dns/dns/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.350237517Z namespaces/openshift-dns/pods/dns-default-f96fd/kube-rbac-proxy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.350243387Z namespaces/openshift-dns/pods/dns-default-f96fd/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.350275738Z namespaces/openshift-dns/pods/dns-default-f96fd/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.350320569Z namespaces/openshift-dns/pods/dns-default-f96fd/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.350418031Z namespaces/openshift-dns/pods/dns-default-f96fd/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.350482892Z namespaces/openshift-dns/pods/dns-default-f96fd/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.350503343Z namespaces/openshift-dns/pods/node-resolver-7qqxm/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.350604785Z namespaces/openshift-dns/pods/node-resolver-7qqxm/node-resolver-7qqxm.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.350687627Z namespaces/openshift-dns/pods/node-resolver-7qqxm/dns-node-resolver/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.350697877Z namespaces/openshift-dns/pods/node-resolver-7qqxm/dns-node-resolver/dns-node-resolver/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.350710107Z namespaces/openshift-dns/pods/node-resolver-7qqxm/dns-node-resolver/dns-node-resolver/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.350804909Z namespaces/openshift-dns/pods/node-resolver-7qqxm/dns-node-resolver/dns-node-resolver/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.350830759Z namespaces/openshift-dns/pods/node-resolver-7qqxm/dns-node-resolver/dns-node-resolver/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.350907121Z namespaces/openshift-dns/pods/node-resolver-7qqxm/dns-node-resolver/dns-node-resolver/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.350962752Z namespaces/openshift-dns/pods/node-resolver-8gwcq/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.351014913Z namespaces/openshift-dns/pods/node-resolver-8gwcq/node-resolver-8gwcq.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.351098255Z namespaces/openshift-dns/pods/node-resolver-8gwcq/dns-node-resolver/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.351107495Z namespaces/openshift-dns/pods/node-resolver-8gwcq/dns-node-resolver/dns-node-resolver/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.351137226Z namespaces/openshift-dns/pods/node-resolver-8gwcq/dns-node-resolver/dns-node-resolver/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.351192687Z namespaces/openshift-dns/pods/node-resolver-8gwcq/dns-node-resolver/dns-node-resolver/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.351278039Z namespaces/openshift-dns/pods/node-resolver-8gwcq/dns-node-resolver/dns-node-resolver/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.35133672Z namespaces/openshift-dns/pods/node-resolver-8gwcq/dns-node-resolver/dns-node-resolver/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.351378401Z namespaces/openshift-dns/pods/node-resolver-gcwqn/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.351410401Z namespaces/openshift-dns/pods/node-resolver-gcwqn/node-resolver-gcwqn.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.351493163Z namespaces/openshift-dns/pods/node-resolver-gcwqn/dns-node-resolver/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.351504013Z namespaces/openshift-dns/pods/node-resolver-gcwqn/dns-node-resolver/dns-node-resolver/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.351509483Z namespaces/openshift-dns/pods/node-resolver-gcwqn/dns-node-resolver/dns-node-resolver/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.351521144Z namespaces/openshift-dns/pods/node-resolver-gcwqn/dns-node-resolver/dns-node-resolver/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.351626035Z namespaces/openshift-dns/pods/node-resolver-gcwqn/dns-node-resolver/dns-node-resolver/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.351702097Z namespaces/openshift-dns/pods/node-resolver-gcwqn/dns-node-resolver/dns-node-resolver/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.351741138Z namespaces/openshift-dns/policy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.351782149Z namespaces/openshift-dns/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.35183912Z namespaces/openshift-dns/route.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.351890941Z namespaces/openshift-dns/route.openshift.io/routes.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.351971282Z namespaces/openshift-etcd/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.352021983Z namespaces/openshift-etcd/openshift-etcd.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.352082805Z namespaces/openshift-etcd/apps.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.352141606Z namespaces/openshift-etcd/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.352213598Z namespaces/openshift-etcd/apps/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.352260418Z namespaces/openshift-etcd/apps/daemonsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.35233993Z namespaces/openshift-etcd/apps/deployments.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.352417382Z namespaces/openshift-etcd/apps/replicasets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.352542454Z namespaces/openshift-etcd/apps/statefulsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.352607055Z namespaces/openshift-etcd/autoscaling/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.352678427Z namespaces/openshift-etcd/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.352715348Z namespaces/openshift-etcd/batch/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.352769329Z namespaces/openshift-etcd/batch/cronjobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.35284407Z namespaces/openshift-etcd/batch/jobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.352901881Z namespaces/openshift-etcd/build.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.353025474Z namespaces/openshift-etcd/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.353110516Z namespaces/openshift-etcd/build.openshift.io/builds.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.353214298Z namespaces/openshift-etcd/core/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.353236288Z namespaces/openshift-etcd/core/configmaps.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.353351701Z namespaces/openshift-etcd/core/endpoints.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.353428832Z namespaces/openshift-etcd/core/events.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.353509624Z namespaces/openshift-etcd/core/persistentvolumeclaims.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.353595126Z namespaces/openshift-etcd/core/pods.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.353670707Z namespaces/openshift-etcd/core/replicationcontrollers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.35380927Z namespaces/openshift-etcd/core/secrets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.353962523Z namespaces/openshift-etcd/core/services.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.354017714Z namespaces/openshift-etcd/discovery.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.354073365Z namespaces/openshift-etcd/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.354105136Z namespaces/openshift-etcd/image.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.354275519Z namespaces/openshift-etcd/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.35430791Z namespaces/openshift-etcd/k8s.ovn.org/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.354361581Z namespaces/openshift-etcd/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.354448923Z namespaces/openshift-etcd/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.354525614Z namespaces/openshift-etcd/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.354601216Z namespaces/openshift-etcd/monitoring.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.354650967Z namespaces/openshift-etcd/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.354713158Z namespaces/openshift-etcd/networking.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.354749829Z namespaces/openshift-etcd/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.35480511Z namespaces/openshift-etcd/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.35481488Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.354864761Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.355008764Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.355216898Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.355345971Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.355747199Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.355930693Z namespaces/openshift-etcd/policy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.355996054Z namespaces/openshift-etcd/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.356049775Z namespaces/openshift-etcd/route.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.356092816Z namespaces/openshift-etcd/route.openshift.io/routes.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.356222899Z namespaces/openshift-host-network/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.35626278Z namespaces/openshift-host-network/openshift-host-network.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.356336461Z namespaces/openshift-host-network/apps.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.356352982Z namespaces/openshift-host-network/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.356427563Z namespaces/openshift-host-network/apps/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.356458674Z namespaces/openshift-host-network/apps/daemonsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.356551446Z namespaces/openshift-host-network/apps/deployments.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.356626217Z namespaces/openshift-host-network/apps/replicasets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.356716719Z namespaces/openshift-host-network/apps/statefulsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.35676871Z namespaces/openshift-host-network/autoscaling/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.356817281Z namespaces/openshift-host-network/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.356875592Z namespaces/openshift-host-network/batch/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.356931713Z namespaces/openshift-host-network/batch/cronjobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.356999704Z namespaces/openshift-host-network/batch/jobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.357056756Z namespaces/openshift-host-network/build.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.357101897Z namespaces/openshift-host-network/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.357205859Z namespaces/openshift-host-network/build.openshift.io/builds.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.35728823Z namespaces/openshift-host-network/core/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.357341691Z namespaces/openshift-host-network/core/configmaps.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.357444264Z namespaces/openshift-host-network/core/endpoints.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.357523345Z namespaces/openshift-host-network/core/events.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.357603677Z namespaces/openshift-host-network/core/persistentvolumeclaims.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.357713599Z namespaces/openshift-host-network/core/pods.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.357796441Z namespaces/openshift-host-network/core/replicationcontrollers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.357946594Z namespaces/openshift-host-network/core/secrets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.358053206Z namespaces/openshift-host-network/core/services.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.358100277Z namespaces/openshift-host-network/core/resourcequotas/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.358183888Z namespaces/openshift-host-network/core/resourcequotas/host-network-namespace-quotas.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.35823643Z namespaces/openshift-host-network/discovery.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.358282621Z namespaces/openshift-host-network/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.358343812Z namespaces/openshift-host-network/image.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.358401793Z namespaces/openshift-host-network/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.358454834Z namespaces/openshift-host-network/k8s.ovn.org/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.358520515Z namespaces/openshift-host-network/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.358624457Z namespaces/openshift-host-network/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.358702729Z namespaces/openshift-host-network/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.35875064Z namespaces/openshift-host-network/monitoring.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.358792481Z namespaces/openshift-host-network/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.358837192Z namespaces/openshift-host-network/networking.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.358885643Z namespaces/openshift-host-network/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.358938204Z namespaces/openshift-host-network/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.358951374Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.359011365Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.359194919Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.359410944Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.359551836Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.359763921Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.359951564Z namespaces/openshift-host-network/policy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.359984555Z namespaces/openshift-host-network/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.360044166Z namespaces/openshift-host-network/route.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.360089917Z namespaces/openshift-host-network/route.openshift.io/routes.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.360185719Z namespaces/openshift-image-registry/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.36023859Z namespaces/openshift-image-registry/openshift-image-registry.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.360296391Z namespaces/openshift-image-registry/apps.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.360356383Z namespaces/openshift-image-registry/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.360393023Z namespaces/openshift-image-registry/apps/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.360434394Z namespaces/openshift-image-registry/apps/daemonsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.360580737Z namespaces/openshift-image-registry/apps/deployments.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.360804532Z namespaces/openshift-image-registry/apps/replicasets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.361005546Z namespaces/openshift-image-registry/apps/statefulsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.361051567Z namespaces/openshift-image-registry/autoscaling/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.361092767Z namespaces/openshift-image-registry/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.361161859Z namespaces/openshift-image-registry/batch/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.36120269Z namespaces/openshift-image-registry/batch/cronjobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.361299202Z namespaces/openshift-image-registry/batch/jobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.361364273Z namespaces/openshift-image-registry/build.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.361433265Z namespaces/openshift-image-registry/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.361529856Z namespaces/openshift-image-registry/build.openshift.io/builds.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.361566427Z namespaces/openshift-image-registry/coordination.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.361572127Z namespaces/openshift-image-registry/coordination.k8s.io/leases/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.361611448Z namespaces/openshift-image-registry/coordination.k8s.io/leases/openshift-master-controllers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.36168738Z namespaces/openshift-image-registry/core/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.361752051Z namespaces/openshift-image-registry/core/configmaps.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.362846163Z namespaces/openshift-image-registry/core/endpoints.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.362949525Z namespaces/openshift-image-registry/core/events.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.363370114Z namespaces/openshift-image-registry/core/persistentvolumeclaims.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.363454895Z namespaces/openshift-image-registry/core/pods.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.36369076Z namespaces/openshift-image-registry/core/replicationcontrollers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.363820633Z namespaces/openshift-image-registry/core/secrets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.364044147Z namespaces/openshift-image-registry/core/services.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.364108119Z namespaces/openshift-image-registry/discovery.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.36417232Z namespaces/openshift-image-registry/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.364249722Z namespaces/openshift-image-registry/image.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.364274392Z namespaces/openshift-image-registry/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.364337684Z namespaces/openshift-image-registry/k8s.ovn.org/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.364452416Z namespaces/openshift-image-registry/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.364535077Z namespaces/openshift-image-registry/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.364625209Z namespaces/openshift-image-registry/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.364692181Z namespaces/openshift-image-registry/monitoring.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.364793913Z namespaces/openshift-image-registry/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.364870424Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.364917705Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/image-registry-operator-alerts.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.365021057Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/image-registry-rules.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.36514627Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/imagestreams-rules.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.365237162Z namespaces/openshift-image-registry/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.365286653Z namespaces/openshift-image-registry/monitoring.coreos.com/servicemonitors/image-registry.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.365332154Z namespaces/openshift-image-registry/networking.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.365401075Z namespaces/openshift-image-registry/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.365437566Z namespaces/openshift-image-registry/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.365474356Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.365509817Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.365700551Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.365905315Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.366048848Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.366291153Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.366603519Z namespaces/openshift-image-registry/pods/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.36661181Z namespaces/openshift-image-registry/pods/image-registry-56565fd7f7-2x2n8/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.366652771Z namespaces/openshift-image-registry/pods/image-registry-56565fd7f7-2x2n8/image-registry-56565fd7f7-2x2n8.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.366768653Z namespaces/openshift-image-registry/pods/image-registry-56565fd7f7-2x2n8/registry/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.366779393Z namespaces/openshift-image-registry/pods/image-registry-56565fd7f7-2x2n8/registry/registry/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.366790673Z namespaces/openshift-image-registry/pods/image-registry-56565fd7f7-2x2n8/registry/registry/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.366856995Z namespaces/openshift-image-registry/pods/image-registry-56565fd7f7-2x2n8/registry/registry/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.36909709Z namespaces/openshift-image-registry/pods/image-registry-56565fd7f7-2x2n8/registry/registry/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.369183742Z namespaces/openshift-image-registry/pods/image-registry-56565fd7f7-2x2n8/registry/registry/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.369204552Z namespaces/openshift-image-registry/pods/node-ca-7nqzl/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.369254323Z namespaces/openshift-image-registry/pods/node-ca-7nqzl/node-ca-7nqzl.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.369330685Z namespaces/openshift-image-registry/pods/node-ca-7nqzl/node-ca/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.369339775Z namespaces/openshift-image-registry/pods/node-ca-7nqzl/node-ca/node-ca/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.369344945Z namespaces/openshift-image-registry/pods/node-ca-7nqzl/node-ca/node-ca/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.369385086Z namespaces/openshift-image-registry/pods/node-ca-7nqzl/node-ca/node-ca/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.369555179Z namespaces/openshift-image-registry/pods/node-ca-7nqzl/node-ca/node-ca/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.369630071Z namespaces/openshift-image-registry/pods/node-ca-7nqzl/node-ca/node-ca/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.369664152Z namespaces/openshift-image-registry/pods/node-ca-wptfc/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.369709962Z namespaces/openshift-image-registry/pods/node-ca-wptfc/node-ca-wptfc.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.369778424Z namespaces/openshift-image-registry/pods/node-ca-wptfc/node-ca/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.369797564Z namespaces/openshift-image-registry/pods/node-ca-wptfc/node-ca/node-ca/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.369805495Z namespaces/openshift-image-registry/pods/node-ca-wptfc/node-ca/node-ca/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.369851185Z namespaces/openshift-image-registry/pods/node-ca-wptfc/node-ca/node-ca/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.370023739Z namespaces/openshift-image-registry/pods/node-ca-wptfc/node-ca/node-ca/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.37008723Z namespaces/openshift-image-registry/pods/node-ca-wptfc/node-ca/node-ca/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.370113091Z namespaces/openshift-image-registry/pods/node-ca-z7pbf/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.370234173Z namespaces/openshift-image-registry/pods/node-ca-z7pbf/node-ca-z7pbf.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.370300234Z namespaces/openshift-image-registry/pods/node-ca-z7pbf/node-ca/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.370309005Z namespaces/openshift-image-registry/pods/node-ca-z7pbf/node-ca/node-ca/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.370315615Z namespaces/openshift-image-registry/pods/node-ca-z7pbf/node-ca/node-ca/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.370367556Z namespaces/openshift-image-registry/pods/node-ca-z7pbf/node-ca/node-ca/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.370541889Z namespaces/openshift-image-registry/pods/node-ca-z7pbf/node-ca/node-ca/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.370625751Z namespaces/openshift-image-registry/pods/node-ca-z7pbf/node-ca/node-ca/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.370775194Z namespaces/openshift-image-registry/policy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.370793075Z namespaces/openshift-image-registry/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.370805035Z namespaces/openshift-image-registry/route.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.370819855Z namespaces/openshift-image-registry/route.openshift.io/routes.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.370845356Z namespaces/openshift-infra/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.370857446Z namespaces/openshift-infra/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.370861836Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.370910577Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.37105317Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.371249084Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.371396327Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.371614121Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.371829876Z namespaces/openshift-ingress-canary/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.371875997Z namespaces/openshift-ingress-canary/openshift-ingress-canary.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.371957098Z namespaces/openshift-ingress-canary/apps.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.371974519Z namespaces/openshift-ingress-canary/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.372046Z namespaces/openshift-ingress-canary/apps/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.372095211Z namespaces/openshift-ingress-canary/apps/daemonsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.372226024Z namespaces/openshift-ingress-canary/apps/deployments.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.372307485Z namespaces/openshift-ingress-canary/apps/replicasets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.372390067Z namespaces/openshift-ingress-canary/apps/statefulsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.372448298Z namespaces/openshift-ingress-canary/autoscaling/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.372488409Z namespaces/openshift-ingress-canary/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.37254946Z namespaces/openshift-ingress-canary/batch/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.372587811Z namespaces/openshift-ingress-canary/batch/cronjobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.372677133Z namespaces/openshift-ingress-canary/batch/jobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.372708493Z namespaces/openshift-ingress-canary/build.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.372753434Z namespaces/openshift-ingress-canary/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.372832146Z namespaces/openshift-ingress-canary/build.openshift.io/builds.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.372912598Z namespaces/openshift-ingress-canary/core/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.372956968Z namespaces/openshift-ingress-canary/core/configmaps.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.373063641Z namespaces/openshift-ingress-canary/core/endpoints.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.373177793Z namespaces/openshift-ingress-canary/core/events.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.373320876Z namespaces/openshift-ingress-canary/core/persistentvolumeclaims.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.373400667Z namespaces/openshift-ingress-canary/core/pods.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.373559291Z namespaces/openshift-ingress-canary/core/replicationcontrollers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.373697573Z namespaces/openshift-ingress-canary/core/secrets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.373891917Z namespaces/openshift-ingress-canary/core/services.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.373944818Z namespaces/openshift-ingress-canary/discovery.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.373999429Z namespaces/openshift-ingress-canary/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.37403973Z namespaces/openshift-ingress-canary/image.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.374079981Z namespaces/openshift-ingress-canary/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.374238704Z namespaces/openshift-ingress-canary/k8s.ovn.org/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.374286215Z namespaces/openshift-ingress-canary/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.374371747Z namespaces/openshift-ingress-canary/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.374453179Z namespaces/openshift-ingress-canary/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.37453018Z namespaces/openshift-ingress-canary/monitoring.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.374576131Z namespaces/openshift-ingress-canary/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.374633682Z namespaces/openshift-ingress-canary/networking.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.374694124Z namespaces/openshift-ingress-canary/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.374785896Z namespaces/openshift-ingress-canary/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.374807396Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.374853057Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.375032661Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.375236275Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.375371897Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.375579312Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.376059431Z namespaces/openshift-ingress-canary/pods/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.376069571Z namespaces/openshift-ingress-canary/pods/ingress-canary-5xnh4/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.376109992Z namespaces/openshift-ingress-canary/pods/ingress-canary-5xnh4/ingress-canary-5xnh4.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.376227075Z namespaces/openshift-ingress-canary/pods/ingress-canary-5xnh4/serve-healthcheck-canary/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.376241715Z namespaces/openshift-ingress-canary/pods/ingress-canary-5xnh4/serve-healthcheck-canary/serve-healthcheck-canary/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.376246775Z namespaces/openshift-ingress-canary/pods/ingress-canary-5xnh4/serve-healthcheck-canary/serve-healthcheck-canary/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.376288246Z namespaces/openshift-ingress-canary/pods/ingress-canary-5xnh4/serve-healthcheck-canary/serve-healthcheck-canary/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.376396428Z namespaces/openshift-ingress-canary/pods/ingress-canary-5xnh4/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.37646295Z namespaces/openshift-ingress-canary/pods/ingress-canary-5xnh4/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.376527191Z namespaces/openshift-ingress-canary/pods/ingress-canary-bbsrj/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.376545851Z namespaces/openshift-ingress-canary/pods/ingress-canary-bbsrj/ingress-canary-bbsrj.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.376651443Z namespaces/openshift-ingress-canary/pods/ingress-canary-bbsrj/serve-healthcheck-canary/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.376664884Z namespaces/openshift-ingress-canary/pods/ingress-canary-bbsrj/serve-healthcheck-canary/serve-healthcheck-canary/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.376671454Z namespaces/openshift-ingress-canary/pods/ingress-canary-bbsrj/serve-healthcheck-canary/serve-healthcheck-canary/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.376709875Z namespaces/openshift-ingress-canary/pods/ingress-canary-bbsrj/serve-healthcheck-canary/serve-healthcheck-canary/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.376804957Z namespaces/openshift-ingress-canary/pods/ingress-canary-bbsrj/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.376876618Z namespaces/openshift-ingress-canary/pods/ingress-canary-bbsrj/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.376910509Z namespaces/openshift-ingress-canary/pods/ingress-canary-sd8qz/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.37696374Z namespaces/openshift-ingress-canary/pods/ingress-canary-sd8qz/ingress-canary-sd8qz.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.377032041Z namespaces/openshift-ingress-canary/pods/ingress-canary-sd8qz/serve-healthcheck-canary/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.377049131Z namespaces/openshift-ingress-canary/pods/ingress-canary-sd8qz/serve-healthcheck-canary/serve-healthcheck-canary/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.377059202Z namespaces/openshift-ingress-canary/pods/ingress-canary-sd8qz/serve-healthcheck-canary/serve-healthcheck-canary/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.377104003Z namespaces/openshift-ingress-canary/pods/ingress-canary-sd8qz/serve-healthcheck-canary/serve-healthcheck-canary/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.377224885Z namespaces/openshift-ingress-canary/pods/ingress-canary-sd8qz/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.377296947Z namespaces/openshift-ingress-canary/pods/ingress-canary-sd8qz/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.377328927Z namespaces/openshift-ingress-canary/policy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.377355198Z namespaces/openshift-ingress-canary/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.377425679Z namespaces/openshift-ingress-canary/route.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.37745216Z namespaces/openshift-ingress-canary/route.openshift.io/routes.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.377560172Z namespaces/openshift-ingress-operator/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.377612833Z namespaces/openshift-ingress-operator/openshift-ingress-operator.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.377685974Z namespaces/openshift-ingress-operator/apps.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.377726835Z namespaces/openshift-ingress-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.377788977Z namespaces/openshift-ingress-operator/apps/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.377842418Z namespaces/openshift-ingress-operator/apps/daemonsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.377925609Z namespaces/openshift-ingress-operator/apps/deployments.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.378004521Z namespaces/openshift-ingress-operator/apps/replicasets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.378083692Z namespaces/openshift-ingress-operator/apps/statefulsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.378142904Z namespaces/openshift-ingress-operator/autoscaling/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.378199655Z namespaces/openshift-ingress-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.378268186Z namespaces/openshift-ingress-operator/batch/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.378297177Z namespaces/openshift-ingress-operator/batch/cronjobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.378392289Z namespaces/openshift-ingress-operator/batch/jobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.378431319Z namespaces/openshift-ingress-operator/build.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.378548232Z namespaces/openshift-ingress-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.378660424Z namespaces/openshift-ingress-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.378740346Z namespaces/openshift-ingress-operator/core/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.378773256Z namespaces/openshift-ingress-operator/core/configmaps.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.379805737Z namespaces/openshift-ingress-operator/core/endpoints.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.379893569Z namespaces/openshift-ingress-operator/core/events.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.380019832Z namespaces/openshift-ingress-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.380099183Z namespaces/openshift-ingress-operator/core/pods.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.380209976Z namespaces/openshift-ingress-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.380352829Z namespaces/openshift-ingress-operator/core/secrets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.380565213Z namespaces/openshift-ingress-operator/core/services.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.380622394Z namespaces/openshift-ingress-operator/discovery.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.380694585Z namespaces/openshift-ingress-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.380762547Z namespaces/openshift-ingress-operator/image.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.380809518Z namespaces/openshift-ingress-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.380863139Z namespaces/openshift-ingress-operator/ingress.operator.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.380872849Z namespaces/openshift-ingress-operator/ingress.operator.openshift.io/dnsrecords/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.38091925Z namespaces/openshift-ingress-operator/ingress.operator.openshift.io/dnsrecords/default-wildcard.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.380984091Z namespaces/openshift-ingress-operator/k8s.ovn.org/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.381026122Z namespaces/openshift-ingress-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.381159925Z namespaces/openshift-ingress-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.381284247Z namespaces/openshift-ingress-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.381347459Z namespaces/openshift-ingress-operator/monitoring.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.38140924Z namespaces/openshift-ingress-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.381470661Z namespaces/openshift-ingress-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.381502642Z namespaces/openshift-ingress-operator/monitoring.coreos.com/prometheusrules/ingress-operator.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.381587363Z namespaces/openshift-ingress-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.381621044Z namespaces/openshift-ingress-operator/monitoring.coreos.com/servicemonitors/ingress-operator.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.381698816Z namespaces/openshift-ingress-operator/networking.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.381726436Z namespaces/openshift-ingress-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.381789808Z namespaces/openshift-ingress-operator/operator.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.381797528Z namespaces/openshift-ingress-operator/operator.openshift.io/ingresscontrollers/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.381839119Z namespaces/openshift-ingress-operator/operator.openshift.io/ingresscontrollers/default.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.38190902Z namespaces/openshift-ingress-operator/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.38192701Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.381982492Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.382223327Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.38240398Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.382561463Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.382809978Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.383017903Z namespaces/openshift-ingress-operator/policy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.383048683Z namespaces/openshift-ingress-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.383135965Z namespaces/openshift-ingress-operator/route.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.383186076Z namespaces/openshift-ingress-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.383297158Z namespaces/openshift-ingress/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.383346019Z namespaces/openshift-ingress/openshift-ingress.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.383418611Z namespaces/openshift-ingress/apps.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.383461992Z namespaces/openshift-ingress/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.383524183Z namespaces/openshift-ingress/apps/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.383569964Z namespaces/openshift-ingress/apps/daemonsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.383661786Z namespaces/openshift-ingress/apps/deployments.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.383911461Z namespaces/openshift-ingress/apps/replicasets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.384114285Z namespaces/openshift-ingress/apps/statefulsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.384190416Z namespaces/openshift-ingress/autoscaling/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.384241267Z namespaces/openshift-ingress/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.384302269Z namespaces/openshift-ingress/batch/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.384346899Z namespaces/openshift-ingress/batch/cronjobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.384439581Z namespaces/openshift-ingress/batch/jobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.384502203Z namespaces/openshift-ingress/build.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.384551024Z namespaces/openshift-ingress/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.384634795Z namespaces/openshift-ingress/build.openshift.io/builds.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.384939991Z namespaces/openshift-ingress/coordination.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.384949972Z namespaces/openshift-ingress/coordination.k8s.io/leases/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.384986143Z namespaces/openshift-ingress/coordination.k8s.io/leases/istio-gateway-ca-openshift-gateway.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.385087425Z namespaces/openshift-ingress/coordination.k8s.io/leases/istio-gateway-deployment-openshift-gateway.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.385191897Z namespaces/openshift-ingress/coordination.k8s.io/leases/istio-gateway-status-leader-openshift-gateway.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.38533355Z namespaces/openshift-ingress/core/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.38537786Z namespaces/openshift-ingress/core/configmaps.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.385788139Z namespaces/openshift-ingress/core/endpoints.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.385895031Z namespaces/openshift-ingress/core/events.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.386085965Z namespaces/openshift-ingress/core/persistentvolumeclaims.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.386222688Z namespaces/openshift-ingress/core/pods.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.386525724Z namespaces/openshift-ingress/core/replicationcontrollers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.386686697Z namespaces/openshift-ingress/core/secrets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.386941672Z namespaces/openshift-ingress/core/services.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.387031844Z namespaces/openshift-ingress/discovery.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.387073065Z namespaces/openshift-ingress/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.387187797Z namespaces/openshift-ingress/gateway.networking.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.387208817Z namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.387227378Z namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/openshift-ai-inference.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.387288149Z namespaces/openshift-ingress/image.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.38733974Z namespaces/openshift-ingress/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.387397951Z namespaces/openshift-ingress/k8s.ovn.org/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.387444852Z namespaces/openshift-ingress/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.387528024Z namespaces/openshift-ingress/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.387609776Z namespaces/openshift-ingress/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.387665547Z namespaces/openshift-ingress/monitoring.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.387698878Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.387771129Z namespaces/openshift-ingress/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.3878004Z namespaces/openshift-ingress/monitoring.coreos.com/podmonitors/openshift-ai-inference-metrics.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.387880041Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.387907722Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors/router-default.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.387977693Z namespaces/openshift-ingress/networking.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.388003504Z namespaces/openshift-ingress/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.388045995Z namespaces/openshift-ingress/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.388059125Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.388117306Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.38829468Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.388488374Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.388761799Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.388998054Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.389220858Z namespaces/openshift-ingress/pods/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.389236099Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-bv8fm/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.389255649Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-bv8fm/istiod-openshift-gateway-5677ccb49b-bv8fm.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.389376292Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-bv8fm/discovery/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.389386352Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-bv8fm/discovery/discovery/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.389399962Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-bv8fm/discovery/discovery/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.389435643Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-bv8fm/discovery/discovery/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.399912645Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-bv8fm/discovery/discovery/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.399974356Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-bv8fm/discovery/discovery/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.400044798Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.400063528Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.400271893Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l/istio-proxy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.400289303Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l/istio-proxy/istio-proxy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.400295693Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.400334674Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.40164261Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-8wm6l/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.401725112Z namespaces/openshift-ingress/pods/router-default-66dc97f78c-hsl5f/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.401759593Z namespaces/openshift-ingress/pods/router-default-66dc97f78c-hsl5f/router-default-66dc97f78c-hsl5f.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.401936056Z namespaces/openshift-ingress/pods/router-default-66dc97f78c-hsl5f/router/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.401945007Z namespaces/openshift-ingress/pods/router-default-66dc97f78c-hsl5f/router/router/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.401957387Z namespaces/openshift-ingress/pods/router-default-66dc97f78c-hsl5f/router/router/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.402012068Z namespaces/openshift-ingress/pods/router-default-66dc97f78c-hsl5f/router/router/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.402147911Z namespaces/openshift-ingress/pods/router-default-66dc97f78c-hsl5f/router/router/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.402287054Z namespaces/openshift-ingress/pods/router-default-66dc97f78c-hsl5f/router/router/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.402320834Z namespaces/openshift-ingress/policy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.402378185Z namespaces/openshift-ingress/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.402438257Z namespaces/openshift-ingress/route.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.402485488Z namespaces/openshift-ingress/route.openshift.io/routes.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.40258981Z namespaces/openshift-insights/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.402638971Z namespaces/openshift-insights/openshift-insights.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.402826045Z namespaces/openshift-insights/apps.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.402856995Z namespaces/openshift-insights/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.402941117Z namespaces/openshift-insights/apps/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.402970068Z namespaces/openshift-insights/apps/daemonsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.40310687Z namespaces/openshift-insights/apps/deployments.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.403242063Z namespaces/openshift-insights/apps/replicasets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.403344835Z namespaces/openshift-insights/apps/statefulsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.403373236Z namespaces/openshift-insights/apps/deployments/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.403435537Z namespaces/openshift-insights/apps/deployments/insights-operator.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.403502728Z namespaces/openshift-insights/autoscaling/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.403531929Z namespaces/openshift-insights/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.40360364Z namespaces/openshift-insights/batch/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.403633261Z namespaces/openshift-insights/batch/cronjobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.403724643Z namespaces/openshift-insights/batch/jobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.403779624Z namespaces/openshift-insights/build.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.403824875Z namespaces/openshift-insights/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.403901006Z namespaces/openshift-insights/build.openshift.io/builds.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.403978078Z namespaces/openshift-insights/core/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.404027789Z namespaces/openshift-insights/core/configmaps.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.405012369Z namespaces/openshift-insights/core/endpoints.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.405151582Z namespaces/openshift-insights/core/events.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.405473868Z namespaces/openshift-insights/core/persistentvolumeclaims.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.405651412Z namespaces/openshift-insights/core/pods.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.405984599Z namespaces/openshift-insights/core/replicationcontrollers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.406106321Z namespaces/openshift-insights/core/secrets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.406359046Z namespaces/openshift-insights/core/services.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.406439658Z namespaces/openshift-insights/core/configmaps/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.40652298Z namespaces/openshift-insights/core/configmaps/service-ca-bundle.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.406614602Z namespaces/openshift-insights/core/serviceaccounts/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.406666012Z namespaces/openshift-insights/core/serviceaccounts/gather.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.406790775Z namespaces/openshift-insights/core/serviceaccounts/operator.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.406854236Z namespaces/openshift-insights/core/services/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.406909907Z namespaces/openshift-insights/core/services/metrics.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.406987049Z namespaces/openshift-insights/discovery.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.40704521Z namespaces/openshift-insights/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.407117962Z namespaces/openshift-insights/image.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.407189773Z namespaces/openshift-insights/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.407259364Z namespaces/openshift-insights/k8s.ovn.org/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.407321746Z namespaces/openshift-insights/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.407444748Z namespaces/openshift-insights/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.407554281Z namespaces/openshift-insights/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.407615032Z namespaces/openshift-insights/monitoring.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.407672023Z namespaces/openshift-insights/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.407761335Z namespaces/openshift-insights/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.407815226Z namespaces/openshift-insights/monitoring.coreos.com/prometheusrules/insights-prometheus-rules.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.407892577Z namespaces/openshift-insights/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.407942889Z namespaces/openshift-insights/monitoring.coreos.com/servicemonitors/insights-operator.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.40802566Z namespaces/openshift-insights/networking.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.408084871Z namespaces/openshift-insights/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.408188474Z namespaces/openshift-insights/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.408208324Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.408249135Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.408479069Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.408704704Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.408886397Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.409141333Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.409439039Z namespaces/openshift-insights/pods/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.409446449Z namespaces/openshift-insights/pods/insights-operator-5885fdb759-vkqmg/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.40950184Z namespaces/openshift-insights/pods/insights-operator-5885fdb759-vkqmg/insights-operator-5885fdb759-vkqmg.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.409625883Z namespaces/openshift-insights/pods/insights-operator-5885fdb759-vkqmg/insights-operator/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.409636223Z namespaces/openshift-insights/pods/insights-operator-5885fdb759-vkqmg/insights-operator/insights-operator/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.409645403Z namespaces/openshift-insights/pods/insights-operator-5885fdb759-vkqmg/insights-operator/insights-operator/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.409709314Z namespaces/openshift-insights/pods/insights-operator-5885fdb759-vkqmg/insights-operator/insights-operator/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.410234995Z namespaces/openshift-insights/pods/insights-operator-5885fdb759-vkqmg/insights-operator/insights-operator/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.410543961Z namespaces/openshift-insights/pods/insights-runtime-extractor-bttvz/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.410591582Z namespaces/openshift-insights/pods/insights-runtime-extractor-bttvz/insights-runtime-extractor-bttvz.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.410724285Z namespaces/openshift-insights/pods/insights-runtime-extractor-bttvz/exporter/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.410734125Z namespaces/openshift-insights/pods/insights-runtime-extractor-bttvz/exporter/exporter/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.410739015Z namespaces/openshift-insights/pods/insights-runtime-extractor-bttvz/exporter/exporter/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.410753555Z namespaces/openshift-insights/pods/insights-runtime-extractor-bttvz/exporter/exporter/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.410870748Z namespaces/openshift-insights/pods/insights-runtime-extractor-bttvz/exporter/exporter/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.410941329Z namespaces/openshift-insights/pods/insights-runtime-extractor-bttvz/exporter/exporter/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.41096514Z namespaces/openshift-insights/pods/insights-runtime-extractor-bttvz/extractor/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.41097292Z namespaces/openshift-insights/pods/insights-runtime-extractor-bttvz/extractor/extractor/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.41098005Z namespaces/openshift-insights/pods/insights-runtime-extractor-bttvz/extractor/extractor/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.411019041Z namespaces/openshift-insights/pods/insights-runtime-extractor-bttvz/extractor/extractor/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.411114303Z namespaces/openshift-insights/pods/insights-runtime-extractor-bttvz/extractor/extractor/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.411215895Z namespaces/openshift-insights/pods/insights-runtime-extractor-bttvz/extractor/extractor/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.411239315Z namespaces/openshift-insights/pods/insights-runtime-extractor-bttvz/kube-rbac-proxy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.411252205Z namespaces/openshift-insights/pods/insights-runtime-extractor-bttvz/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.411265306Z namespaces/openshift-insights/pods/insights-runtime-extractor-bttvz/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.411354778Z namespaces/openshift-insights/pods/insights-runtime-extractor-bttvz/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.411449909Z namespaces/openshift-insights/pods/insights-runtime-extractor-bttvz/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.411524401Z namespaces/openshift-insights/pods/insights-runtime-extractor-bttvz/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.411573462Z namespaces/openshift-insights/pods/insights-runtime-extractor-gct9w/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.411616953Z namespaces/openshift-insights/pods/insights-runtime-extractor-gct9w/insights-runtime-extractor-gct9w.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.411701645Z namespaces/openshift-insights/pods/insights-runtime-extractor-gct9w/exporter/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.411715295Z namespaces/openshift-insights/pods/insights-runtime-extractor-gct9w/exporter/exporter/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.411721785Z namespaces/openshift-insights/pods/insights-runtime-extractor-gct9w/exporter/exporter/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.411777466Z namespaces/openshift-insights/pods/insights-runtime-extractor-gct9w/exporter/exporter/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.411870528Z namespaces/openshift-insights/pods/insights-runtime-extractor-gct9w/exporter/exporter/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.41193687Z namespaces/openshift-insights/pods/insights-runtime-extractor-gct9w/exporter/exporter/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.41197226Z namespaces/openshift-insights/pods/insights-runtime-extractor-gct9w/extractor/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.411990291Z namespaces/openshift-insights/pods/insights-runtime-extractor-gct9w/extractor/extractor/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.411996701Z namespaces/openshift-insights/pods/insights-runtime-extractor-gct9w/extractor/extractor/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.412016641Z namespaces/openshift-insights/pods/insights-runtime-extractor-gct9w/extractor/extractor/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.412187155Z namespaces/openshift-insights/pods/insights-runtime-extractor-gct9w/extractor/extractor/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.412256426Z namespaces/openshift-insights/pods/insights-runtime-extractor-gct9w/extractor/extractor/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.412300857Z namespaces/openshift-insights/pods/insights-runtime-extractor-gct9w/kube-rbac-proxy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.412310447Z namespaces/openshift-insights/pods/insights-runtime-extractor-gct9w/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.412314667Z namespaces/openshift-insights/pods/insights-runtime-extractor-gct9w/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.412362068Z namespaces/openshift-insights/pods/insights-runtime-extractor-gct9w/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.41245372Z namespaces/openshift-insights/pods/insights-runtime-extractor-gct9w/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.412518671Z namespaces/openshift-insights/pods/insights-runtime-extractor-gct9w/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.412568342Z namespaces/openshift-insights/pods/insights-runtime-extractor-zp6vz/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.412596403Z namespaces/openshift-insights/pods/insights-runtime-extractor-zp6vz/insights-runtime-extractor-zp6vz.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.412695065Z namespaces/openshift-insights/pods/insights-runtime-extractor-zp6vz/exporter/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.412702915Z namespaces/openshift-insights/pods/insights-runtime-extractor-zp6vz/exporter/exporter/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.412706945Z namespaces/openshift-insights/pods/insights-runtime-extractor-zp6vz/exporter/exporter/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.412757006Z namespaces/openshift-insights/pods/insights-runtime-extractor-zp6vz/exporter/exporter/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.412839968Z namespaces/openshift-insights/pods/insights-runtime-extractor-zp6vz/exporter/exporter/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.412905029Z namespaces/openshift-insights/pods/insights-runtime-extractor-zp6vz/exporter/exporter/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.41293199Z namespaces/openshift-insights/pods/insights-runtime-extractor-zp6vz/extractor/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.41293887Z namespaces/openshift-insights/pods/insights-runtime-extractor-zp6vz/extractor/extractor/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.4129487Z namespaces/openshift-insights/pods/insights-runtime-extractor-zp6vz/extractor/extractor/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.412994751Z namespaces/openshift-insights/pods/insights-runtime-extractor-zp6vz/extractor/extractor/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.413093983Z namespaces/openshift-insights/pods/insights-runtime-extractor-zp6vz/extractor/extractor/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.413190835Z namespaces/openshift-insights/pods/insights-runtime-extractor-zp6vz/extractor/extractor/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.413213195Z namespaces/openshift-insights/pods/insights-runtime-extractor-zp6vz/kube-rbac-proxy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.413218595Z namespaces/openshift-insights/pods/insights-runtime-extractor-zp6vz/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.413223796Z namespaces/openshift-insights/pods/insights-runtime-extractor-zp6vz/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.413269996Z namespaces/openshift-insights/pods/insights-runtime-extractor-zp6vz/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.413374919Z namespaces/openshift-insights/pods/insights-runtime-extractor-zp6vz/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.41343598Z namespaces/openshift-insights/pods/insights-runtime-extractor-zp6vz/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.413486941Z namespaces/openshift-insights/policy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.413502941Z namespaces/openshift-insights/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.413575533Z namespaces/openshift-insights/route.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.413615333Z namespaces/openshift-insights/route.openshift.io/routes.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.413677525Z namespaces/openshift-keda/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.413687935Z namespaces/openshift-keda/coordination.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.413693415Z namespaces/openshift-keda/coordination.k8s.io/leases/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.413753696Z namespaces/openshift-keda/coordination.k8s.io/leases/olm-operator.keda.sh.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.413844018Z namespaces/openshift-keda/coordination.k8s.io/leases/operator.keda.sh.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.41392878Z namespaces/openshift-keda/monitoring.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.41393958Z namespaces/openshift-keda/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.413968881Z namespaces/openshift-keda/monitoring.coreos.com/podmonitors/keda-olm-operator.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.414036082Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.414062382Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/keda-admission-webhooks.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.414185505Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/keda-metrics-apiserver.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.414271507Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/keda-operator.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.414328328Z namespaces/openshift-keda/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.414340658Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.414396489Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.414552512Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.414973251Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/custom-metrics-autoscaler.v2.19.0-2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.415202676Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.41540966Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.415565103Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.415744367Z namespaces/openshift-keda/operators.coreos.com/installplans/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.415776447Z namespaces/openshift-keda/operators.coreos.com/installplans/install-zqlvw.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.41589505Z namespaces/openshift-keda/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.415940491Z namespaces/openshift-keda/operators.coreos.com/operatorconditions/custom-metrics-autoscaler.v2.19.0-2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.415993562Z namespaces/openshift-keda/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.416043873Z namespaces/openshift-keda/operators.coreos.com/operatorgroups/openshift-keda.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.416119284Z namespaces/openshift-keda/operators.coreos.com/subscriptions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.416163345Z namespaces/openshift-keda/operators.coreos.com/subscriptions/openshift-custom-metrics-autoscaler-operator.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.416277218Z namespaces/openshift-kube-apiserver-operator/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.416296588Z namespaces/openshift-kube-apiserver-operator/openshift-kube-apiserver-operator.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.4163758Z namespaces/openshift-kube-apiserver-operator/apps.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.416435801Z namespaces/openshift-kube-apiserver-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.416500142Z namespaces/openshift-kube-apiserver-operator/apps/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.416534223Z namespaces/openshift-kube-apiserver-operator/apps/daemonsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.416634035Z namespaces/openshift-kube-apiserver-operator/apps/deployments.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.416705746Z namespaces/openshift-kube-apiserver-operator/apps/replicasets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.416790338Z namespaces/openshift-kube-apiserver-operator/apps/statefulsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.416842529Z namespaces/openshift-kube-apiserver-operator/autoscaling/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.41688883Z namespaces/openshift-kube-apiserver-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.416954111Z namespaces/openshift-kube-apiserver-operator/batch/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.416985112Z namespaces/openshift-kube-apiserver-operator/batch/cronjobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.417083864Z namespaces/openshift-kube-apiserver-operator/batch/jobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.417140625Z namespaces/openshift-kube-apiserver-operator/build.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.417238847Z namespaces/openshift-kube-apiserver-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.417333399Z namespaces/openshift-kube-apiserver-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.417412681Z namespaces/openshift-kube-apiserver-operator/core/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.417465751Z namespaces/openshift-kube-apiserver-operator/core/configmaps.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.417569134Z namespaces/openshift-kube-apiserver-operator/core/endpoints.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.417673406Z namespaces/openshift-kube-apiserver-operator/core/events.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.417774078Z namespaces/openshift-kube-apiserver-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.41785695Z namespaces/openshift-kube-apiserver-operator/core/pods.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.417945171Z namespaces/openshift-kube-apiserver-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.418099724Z namespaces/openshift-kube-apiserver-operator/core/secrets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.418352129Z namespaces/openshift-kube-apiserver-operator/core/services.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.418401421Z namespaces/openshift-kube-apiserver-operator/discovery.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.418440801Z namespaces/openshift-kube-apiserver-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.418490972Z namespaces/openshift-kube-apiserver-operator/image.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.418530023Z namespaces/openshift-kube-apiserver-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.418595634Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.418612705Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.418728937Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.418803929Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.41885596Z namespaces/openshift-kube-apiserver-operator/monitoring.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.41887351Z namespaces/openshift-kube-apiserver-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.418938332Z namespaces/openshift-kube-apiserver-operator/networking.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.418986762Z namespaces/openshift-kube-apiserver-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.419031153Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.419047044Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.419081684Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.419264438Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.419450432Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.419579065Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.419781049Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.419965872Z namespaces/openshift-kube-apiserver-operator/policy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.419992353Z namespaces/openshift-kube-apiserver-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.420050944Z namespaces/openshift-kube-apiserver-operator/route.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.420098055Z namespaces/openshift-kube-apiserver-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.420193367Z namespaces/openshift-kube-apiserver/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.420243218Z namespaces/openshift-kube-apiserver/openshift-kube-apiserver.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.420402481Z namespaces/openshift-kube-apiserver/apps.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.420442362Z namespaces/openshift-kube-apiserver/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.420492123Z namespaces/openshift-kube-apiserver/apps/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.420538184Z namespaces/openshift-kube-apiserver/apps/daemonsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.420636626Z namespaces/openshift-kube-apiserver/apps/deployments.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.420702437Z namespaces/openshift-kube-apiserver/apps/replicasets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.420785339Z namespaces/openshift-kube-apiserver/apps/statefulsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.42082829Z namespaces/openshift-kube-apiserver/autoscaling/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.420871401Z namespaces/openshift-kube-apiserver/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.420924072Z namespaces/openshift-kube-apiserver/batch/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.420976713Z namespaces/openshift-kube-apiserver/batch/cronjobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.421059695Z namespaces/openshift-kube-apiserver/batch/jobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.421094985Z namespaces/openshift-kube-apiserver/build.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.421164887Z namespaces/openshift-kube-apiserver/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.421262449Z namespaces/openshift-kube-apiserver/build.openshift.io/builds.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.421301109Z namespaces/openshift-kube-apiserver/core/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.421363231Z namespaces/openshift-kube-apiserver/core/configmaps.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.421453952Z namespaces/openshift-kube-apiserver/core/endpoints.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.421535474Z namespaces/openshift-kube-apiserver/core/events.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.421614676Z namespaces/openshift-kube-apiserver/core/persistentvolumeclaims.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.421693628Z namespaces/openshift-kube-apiserver/core/pods.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.421771769Z namespaces/openshift-kube-apiserver/core/replicationcontrollers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.421944882Z namespaces/openshift-kube-apiserver/core/secrets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.422142717Z namespaces/openshift-kube-apiserver/core/services.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.422218958Z namespaces/openshift-kube-apiserver/discovery.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.422272949Z namespaces/openshift-kube-apiserver/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.42233621Z namespaces/openshift-kube-apiserver/image.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.422383021Z namespaces/openshift-kube-apiserver/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.422438692Z namespaces/openshift-kube-apiserver/k8s.ovn.org/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.422483633Z namespaces/openshift-kube-apiserver/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.422588686Z namespaces/openshift-kube-apiserver/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.422682137Z namespaces/openshift-kube-apiserver/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.422755129Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.42279981Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.422862881Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.422890862Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/prometheusrules/api-usage.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.423036455Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/prometheusrules/podsecurity.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.423143487Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.423186928Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/servicemonitors/openshift-kube-apiserver.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.423253149Z namespaces/openshift-kube-apiserver/networking.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.423278949Z namespaces/openshift-kube-apiserver/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.423337651Z namespaces/openshift-kube-apiserver/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.423347051Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.423422412Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.423574075Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.423763499Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.423893492Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.424099206Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.42431053Z namespaces/openshift-kube-apiserver/policy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.424347741Z namespaces/openshift-kube-apiserver/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.424401992Z namespaces/openshift-kube-apiserver/route.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.424428493Z namespaces/openshift-kube-apiserver/route.openshift.io/routes.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.424513025Z namespaces/openshift-kube-controller-manager-operator/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.424547855Z namespaces/openshift-kube-controller-manager-operator/openshift-kube-controller-manager-operator.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.424634627Z namespaces/openshift-kube-controller-manager-operator/apps.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.424651757Z namespaces/openshift-kube-controller-manager-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.424746639Z namespaces/openshift-kube-controller-manager-operator/apps/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.4247712Z namespaces/openshift-kube-controller-manager-operator/apps/daemonsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.424864442Z namespaces/openshift-kube-controller-manager-operator/apps/deployments.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.424941803Z namespaces/openshift-kube-controller-manager-operator/apps/replicasets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.425024605Z namespaces/openshift-kube-controller-manager-operator/apps/statefulsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.425059176Z namespaces/openshift-kube-controller-manager-operator/autoscaling/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.425109367Z namespaces/openshift-kube-controller-manager-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.425172618Z namespaces/openshift-kube-controller-manager-operator/batch/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.425230369Z namespaces/openshift-kube-controller-manager-operator/batch/cronjobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.425314511Z namespaces/openshift-kube-controller-manager-operator/batch/jobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.425365822Z namespaces/openshift-kube-controller-manager-operator/build.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.425419933Z namespaces/openshift-kube-controller-manager-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.425496454Z namespaces/openshift-kube-controller-manager-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.425572756Z namespaces/openshift-kube-controller-manager-operator/core/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.425616867Z namespaces/openshift-kube-controller-manager-operator/core/configmaps.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.425725949Z namespaces/openshift-kube-controller-manager-operator/core/endpoints.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.425833431Z namespaces/openshift-kube-controller-manager-operator/core/events.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.425915863Z namespaces/openshift-kube-controller-manager-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.425992884Z namespaces/openshift-kube-controller-manager-operator/core/pods.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.426074156Z namespaces/openshift-kube-controller-manager-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.426315401Z namespaces/openshift-kube-controller-manager-operator/core/secrets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.426535496Z namespaces/openshift-kube-controller-manager-operator/core/services.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.426636448Z namespaces/openshift-kube-controller-manager-operator/discovery.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.426682949Z namespaces/openshift-kube-controller-manager-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.42674499Z namespaces/openshift-kube-controller-manager-operator/image.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.42677272Z namespaces/openshift-kube-controller-manager-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.426855022Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.426882593Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.426976805Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.427056666Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.427106827Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.427180999Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.42725103Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.427282041Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/servicemonitors/kube-controller-manager-operator.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.427343062Z namespaces/openshift-kube-controller-manager-operator/networking.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.427379383Z namespaces/openshift-kube-controller-manager-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.427433214Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.427447674Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.427478365Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.427632738Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.427806692Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.427941284Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.428272731Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.428450614Z namespaces/openshift-kube-controller-manager-operator/policy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.428477145Z namespaces/openshift-kube-controller-manager-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.428540526Z namespaces/openshift-kube-controller-manager-operator/route.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.428568407Z namespaces/openshift-kube-controller-manager-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.428642469Z namespaces/openshift-kube-controller-manager/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.428683979Z namespaces/openshift-kube-controller-manager/openshift-kube-controller-manager.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.428744591Z namespaces/openshift-kube-controller-manager/apps.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.428795061Z namespaces/openshift-kube-controller-manager/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.428852853Z namespaces/openshift-kube-controller-manager/apps/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.428878833Z namespaces/openshift-kube-controller-manager/apps/daemonsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.429071087Z namespaces/openshift-kube-controller-manager/apps/deployments.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.42918842Z namespaces/openshift-kube-controller-manager/apps/replicasets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.429281102Z namespaces/openshift-kube-controller-manager/apps/statefulsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.429331612Z namespaces/openshift-kube-controller-manager/autoscaling/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.429372183Z namespaces/openshift-kube-controller-manager/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.429425874Z namespaces/openshift-kube-controller-manager/batch/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.429453335Z namespaces/openshift-kube-controller-manager/batch/cronjobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.429540517Z namespaces/openshift-kube-controller-manager/batch/jobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.429591838Z namespaces/openshift-kube-controller-manager/build.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.429648299Z namespaces/openshift-kube-controller-manager/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.42971489Z namespaces/openshift-kube-controller-manager/build.openshift.io/builds.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.429768031Z namespaces/openshift-kube-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.429775081Z namespaces/openshift-kube-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.429811452Z namespaces/openshift-kube-controller-manager/coordination.k8s.io/leases/cluster-policy-controller-lock.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.429887394Z namespaces/openshift-kube-controller-manager/core/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.429940395Z namespaces/openshift-kube-controller-manager/core/configmaps.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.430106378Z namespaces/openshift-kube-controller-manager/core/endpoints.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.4302059Z namespaces/openshift-kube-controller-manager/core/events.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.430628039Z namespaces/openshift-kube-controller-manager/core/persistentvolumeclaims.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.43071019Z namespaces/openshift-kube-controller-manager/core/pods.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.430786942Z namespaces/openshift-kube-controller-manager/core/replicationcontrollers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.430928585Z namespaces/openshift-kube-controller-manager/core/secrets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.431089058Z namespaces/openshift-kube-controller-manager/core/services.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.4311691Z namespaces/openshift-kube-controller-manager/discovery.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.431216821Z namespaces/openshift-kube-controller-manager/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.431272752Z namespaces/openshift-kube-controller-manager/image.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.431289732Z namespaces/openshift-kube-controller-manager/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.431374604Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.431413075Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.431522787Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.431603698Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.431635559Z namespaces/openshift-kube-controller-manager/monitoring.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.43168954Z namespaces/openshift-kube-controller-manager/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.431735851Z namespaces/openshift-kube-controller-manager/networking.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.431759062Z namespaces/openshift-kube-controller-manager/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.431821103Z namespaces/openshift-kube-controller-manager/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.431834053Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.431844543Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.432011757Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.432227651Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.432379804Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.432591549Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.432779662Z namespaces/openshift-kube-controller-manager/policy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.432815823Z namespaces/openshift-kube-controller-manager/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.432856404Z namespaces/openshift-kube-controller-manager/route.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.432914605Z namespaces/openshift-kube-controller-manager/route.openshift.io/routes.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.432973436Z namespaces/openshift-kube-scheduler-operator/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.433014317Z namespaces/openshift-kube-scheduler-operator/openshift-kube-scheduler-operator.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.433083199Z namespaces/openshift-kube-scheduler-operator/apps.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.433109859Z namespaces/openshift-kube-scheduler-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.433203641Z namespaces/openshift-kube-scheduler-operator/apps/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.433257022Z namespaces/openshift-kube-scheduler-operator/apps/daemonsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.433367194Z namespaces/openshift-kube-scheduler-operator/apps/deployments.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.433459516Z namespaces/openshift-kube-scheduler-operator/apps/replicasets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.433554248Z namespaces/openshift-kube-scheduler-operator/apps/statefulsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.433599399Z namespaces/openshift-kube-scheduler-operator/autoscaling/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.43362821Z namespaces/openshift-kube-scheduler-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.433690211Z namespaces/openshift-kube-scheduler-operator/batch/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.433717061Z namespaces/openshift-kube-scheduler-operator/batch/cronjobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.433817494Z namespaces/openshift-kube-scheduler-operator/batch/jobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.433883305Z namespaces/openshift-kube-scheduler-operator/build.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.433939716Z namespaces/openshift-kube-scheduler-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.43413681Z namespaces/openshift-kube-scheduler-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.434282143Z namespaces/openshift-kube-scheduler-operator/core/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.434335334Z namespaces/openshift-kube-scheduler-operator/core/configmaps.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.434456287Z namespaces/openshift-kube-scheduler-operator/core/endpoints.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.434535768Z namespaces/openshift-kube-scheduler-operator/core/events.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.43463054Z namespaces/openshift-kube-scheduler-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.434707692Z namespaces/openshift-kube-scheduler-operator/core/pods.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.434784763Z namespaces/openshift-kube-scheduler-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.434903645Z namespaces/openshift-kube-scheduler-operator/core/secrets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.435070669Z namespaces/openshift-kube-scheduler-operator/core/services.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.43511577Z namespaces/openshift-kube-scheduler-operator/discovery.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.435178311Z namespaces/openshift-kube-scheduler-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.435232182Z namespaces/openshift-kube-scheduler-operator/image.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.435261423Z namespaces/openshift-kube-scheduler-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.435325304Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.435355195Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.435455247Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.435526118Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.435561269Z namespaces/openshift-kube-scheduler-operator/monitoring.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.43560654Z namespaces/openshift-kube-scheduler-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.435652211Z namespaces/openshift-kube-scheduler-operator/networking.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.435680551Z namespaces/openshift-kube-scheduler-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.435736092Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.435743873Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.435794394Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.435950597Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.436144521Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.436279473Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.436479508Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.436667021Z namespaces/openshift-kube-scheduler-operator/policy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.436700052Z namespaces/openshift-kube-scheduler-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.436761983Z namespaces/openshift-kube-scheduler-operator/route.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.436794404Z namespaces/openshift-kube-scheduler-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.436873716Z namespaces/openshift-kube-scheduler/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.436920457Z namespaces/openshift-kube-scheduler/openshift-kube-scheduler.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.436973027Z namespaces/openshift-kube-scheduler/apps.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.437002288Z namespaces/openshift-kube-scheduler/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.437046719Z namespaces/openshift-kube-scheduler/apps/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.43707977Z namespaces/openshift-kube-scheduler/apps/daemonsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.437187562Z namespaces/openshift-kube-scheduler/apps/deployments.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.437262613Z namespaces/openshift-kube-scheduler/apps/replicasets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.437344655Z namespaces/openshift-kube-scheduler/apps/statefulsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.437373566Z namespaces/openshift-kube-scheduler/autoscaling/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.437425897Z namespaces/openshift-kube-scheduler/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.437478678Z namespaces/openshift-kube-scheduler/batch/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.437518009Z namespaces/openshift-kube-scheduler/batch/cronjobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.43759664Z namespaces/openshift-kube-scheduler/batch/jobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.437643491Z namespaces/openshift-kube-scheduler/build.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.437686212Z namespaces/openshift-kube-scheduler/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.437777154Z namespaces/openshift-kube-scheduler/build.openshift.io/builds.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.437851815Z namespaces/openshift-kube-scheduler/core/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.437902576Z namespaces/openshift-kube-scheduler/core/configmaps.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.438006779Z namespaces/openshift-kube-scheduler/core/endpoints.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.43808307Z namespaces/openshift-kube-scheduler/core/events.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.438191242Z namespaces/openshift-kube-scheduler/core/persistentvolumeclaims.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.438276624Z namespaces/openshift-kube-scheduler/core/pods.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.438372706Z namespaces/openshift-kube-scheduler/core/replicationcontrollers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.438523319Z namespaces/openshift-kube-scheduler/core/secrets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.438706953Z namespaces/openshift-kube-scheduler/core/services.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.438744313Z namespaces/openshift-kube-scheduler/discovery.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.438809655Z namespaces/openshift-kube-scheduler/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.438858696Z namespaces/openshift-kube-scheduler/image.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.438886246Z namespaces/openshift-kube-scheduler/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.438953098Z namespaces/openshift-kube-scheduler/k8s.ovn.org/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.439002549Z namespaces/openshift-kube-scheduler/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.43907615Z namespaces/openshift-kube-scheduler/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.439190832Z namespaces/openshift-kube-scheduler/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.439245744Z namespaces/openshift-kube-scheduler/monitoring.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.439303705Z namespaces/openshift-kube-scheduler/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.439352766Z namespaces/openshift-kube-scheduler/networking.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.439388067Z namespaces/openshift-kube-scheduler/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.439441308Z namespaces/openshift-kube-scheduler/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.439449228Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.439484769Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.439631961Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.439844336Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.4400405Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.440300555Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.440596151Z namespaces/openshift-kube-scheduler/policy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.440641232Z namespaces/openshift-kube-scheduler/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.440691543Z namespaces/openshift-kube-scheduler/route.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.440744014Z namespaces/openshift-kube-scheduler/route.openshift.io/routes.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.440815856Z namespaces/openshift-kube-storage-version-migrator-operator/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.440857206Z namespaces/openshift-kube-storage-version-migrator-operator/openshift-kube-storage-version-migrator-operator.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.440945478Z namespaces/openshift-kube-storage-version-migrator-operator/apps.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.440996219Z namespaces/openshift-kube-storage-version-migrator-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.441083241Z namespaces/openshift-kube-storage-version-migrator-operator/apps/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.441115011Z namespaces/openshift-kube-storage-version-migrator-operator/apps/daemonsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.441273905Z namespaces/openshift-kube-storage-version-migrator-operator/apps/deployments.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.441425718Z namespaces/openshift-kube-storage-version-migrator-operator/apps/replicasets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.44152931Z namespaces/openshift-kube-storage-version-migrator-operator/apps/statefulsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.441567111Z namespaces/openshift-kube-storage-version-migrator-operator/autoscaling/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.441616752Z namespaces/openshift-kube-storage-version-migrator-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.441666073Z namespaces/openshift-kube-storage-version-migrator-operator/batch/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.441717024Z namespaces/openshift-kube-storage-version-migrator-operator/batch/cronjobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.441785515Z namespaces/openshift-kube-storage-version-migrator-operator/batch/jobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.441840316Z namespaces/openshift-kube-storage-version-migrator-operator/build.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.441885767Z namespaces/openshift-kube-storage-version-migrator-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.441968399Z namespaces/openshift-kube-storage-version-migrator-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.442000989Z namespaces/openshift-kube-storage-version-migrator-operator/coordination.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.44200856Z namespaces/openshift-kube-storage-version-migrator-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.442077511Z namespaces/openshift-kube-storage-version-migrator-operator/coordination.k8s.io/leases/openshift-kube-storage-version-migrator-operator-lock.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.442138152Z namespaces/openshift-kube-storage-version-migrator-operator/core/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.442247654Z namespaces/openshift-kube-storage-version-migrator-operator/core/configmaps.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.442350557Z namespaces/openshift-kube-storage-version-migrator-operator/core/endpoints.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.442445949Z namespaces/openshift-kube-storage-version-migrator-operator/core/events.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.442624202Z namespaces/openshift-kube-storage-version-migrator-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.442702554Z namespaces/openshift-kube-storage-version-migrator-operator/core/pods.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.442825726Z namespaces/openshift-kube-storage-version-migrator-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.442961129Z namespaces/openshift-kube-storage-version-migrator-operator/core/secrets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.443195924Z namespaces/openshift-kube-storage-version-migrator-operator/core/services.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.443250025Z namespaces/openshift-kube-storage-version-migrator-operator/discovery.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.443299936Z namespaces/openshift-kube-storage-version-migrator-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.443351467Z namespaces/openshift-kube-storage-version-migrator-operator/image.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.443411998Z namespaces/openshift-kube-storage-version-migrator-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.443464159Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.44351701Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.443588192Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.443670313Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.443725144Z namespaces/openshift-kube-storage-version-migrator-operator/monitoring.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.443757165Z namespaces/openshift-kube-storage-version-migrator-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.443818516Z namespaces/openshift-kube-storage-version-migrator-operator/networking.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.443873727Z namespaces/openshift-kube-storage-version-migrator-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.443937139Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.443947729Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.44398725Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.444146073Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.444340097Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.4444747Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.444679234Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.444878848Z namespaces/openshift-kube-storage-version-migrator-operator/pods/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.444890518Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6f4d6fb6bc-q46xf/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.444907728Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6f4d6fb6bc-q46xf/kube-storage-version-migrator-operator-6f4d6fb6bc-q46xf.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.445064422Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6f4d6fb6bc-q46xf/kube-storage-version-migrator-operator/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.445075562Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6f4d6fb6bc-q46xf/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.445091412Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6f4d6fb6bc-q46xf/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.445190234Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6f4d6fb6bc-q46xf/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.445313947Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6f4d6fb6bc-q46xf/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.44544503Z namespaces/openshift-kube-storage-version-migrator-operator/policy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.44546339Z namespaces/openshift-kube-storage-version-migrator-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.445524211Z namespaces/openshift-kube-storage-version-migrator-operator/route.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.445539551Z namespaces/openshift-kube-storage-version-migrator-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.445636403Z namespaces/openshift-kube-storage-version-migrator/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.445682674Z namespaces/openshift-kube-storage-version-migrator/openshift-kube-storage-version-migrator.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.445774626Z namespaces/openshift-kube-storage-version-migrator/apps.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.445787956Z namespaces/openshift-kube-storage-version-migrator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.445857208Z namespaces/openshift-kube-storage-version-migrator/apps/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.445894539Z namespaces/openshift-kube-storage-version-migrator/apps/daemonsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.44598421Z namespaces/openshift-kube-storage-version-migrator/apps/deployments.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.446115953Z namespaces/openshift-kube-storage-version-migrator/apps/replicasets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.446249166Z namespaces/openshift-kube-storage-version-migrator/apps/statefulsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.446292047Z namespaces/openshift-kube-storage-version-migrator/autoscaling/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.446349718Z namespaces/openshift-kube-storage-version-migrator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.446403489Z namespaces/openshift-kube-storage-version-migrator/batch/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.44646764Z namespaces/openshift-kube-storage-version-migrator/batch/cronjobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.446579522Z namespaces/openshift-kube-storage-version-migrator/batch/jobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.446631074Z namespaces/openshift-kube-storage-version-migrator/build.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.446687975Z namespaces/openshift-kube-storage-version-migrator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.446760956Z namespaces/openshift-kube-storage-version-migrator/build.openshift.io/builds.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.446810487Z namespaces/openshift-kube-storage-version-migrator/core/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.446843478Z namespaces/openshift-kube-storage-version-migrator/core/configmaps.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.446997111Z namespaces/openshift-kube-storage-version-migrator/core/endpoints.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.447116453Z namespaces/openshift-kube-storage-version-migrator/core/events.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.447273167Z namespaces/openshift-kube-storage-version-migrator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.447378619Z namespaces/openshift-kube-storage-version-migrator/core/pods.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.447516661Z namespaces/openshift-kube-storage-version-migrator/core/replicationcontrollers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.447662984Z namespaces/openshift-kube-storage-version-migrator/core/secrets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.447856598Z namespaces/openshift-kube-storage-version-migrator/core/services.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.448056983Z namespaces/openshift-kube-storage-version-migrator/discovery.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.448102603Z namespaces/openshift-kube-storage-version-migrator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.448169785Z namespaces/openshift-kube-storage-version-migrator/image.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.448187515Z namespaces/openshift-kube-storage-version-migrator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.448256676Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.448297367Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.448377699Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.44845292Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.448509901Z namespaces/openshift-kube-storage-version-migrator/monitoring.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.448537062Z namespaces/openshift-kube-storage-version-migrator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.448599513Z namespaces/openshift-kube-storage-version-migrator/networking.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.448628314Z namespaces/openshift-kube-storage-version-migrator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.448684805Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.448695375Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.448719096Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.448878749Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.449078943Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.449223316Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.44942984Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.449614764Z namespaces/openshift-kube-storage-version-migrator/pods/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.449627804Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-xrcn8/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.449650275Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-xrcn8/migrator-6865c9967-xrcn8.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.449771817Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-xrcn8/graceful-termination/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.449782587Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-xrcn8/graceful-termination/graceful-termination/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.449794798Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-xrcn8/graceful-termination/graceful-termination/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.449824908Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-xrcn8/graceful-termination/graceful-termination/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.44992005Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-xrcn8/graceful-termination/graceful-termination/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.449984971Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-xrcn8/graceful-termination/graceful-termination/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.450022822Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-xrcn8/migrator/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.450032243Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-xrcn8/migrator/migrator/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.450036363Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-xrcn8/migrator/migrator/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.450055733Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-xrcn8/migrator/migrator/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.450181875Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-xrcn8/migrator/migrator/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.450247467Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-xrcn8/migrator/migrator/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.450299578Z namespaces/openshift-kube-storage-version-migrator/policy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.450313928Z namespaces/openshift-kube-storage-version-migrator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.45037213Z namespaces/openshift-kube-storage-version-migrator/route.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.45042478Z namespaces/openshift-kube-storage-version-migrator/route.openshift.io/routes.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.450477561Z namespaces/openshift-lws-operator/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.450533043Z namespaces/openshift-lws-operator/openshift-lws-operator.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.450606994Z namespaces/openshift-lws-operator/apps.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.450626635Z namespaces/openshift-lws-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.450697306Z namespaces/openshift-lws-operator/apps/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.450754307Z namespaces/openshift-lws-operator/apps/daemonsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.450840829Z namespaces/openshift-lws-operator/apps/deployments.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.451001352Z namespaces/openshift-lws-operator/apps/replicasets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.451148705Z namespaces/openshift-lws-operator/apps/statefulsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.451283818Z namespaces/openshift-lws-operator/autoscaling/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.451298008Z namespaces/openshift-lws-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.45136037Z namespaces/openshift-lws-operator/batch/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.45140247Z namespaces/openshift-lws-operator/batch/cronjobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.451485822Z namespaces/openshift-lws-operator/batch/jobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.451520543Z namespaces/openshift-lws-operator/build.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.451564674Z namespaces/openshift-lws-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.451642895Z namespaces/openshift-lws-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.451699196Z namespaces/openshift-lws-operator/coordination.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.451710546Z namespaces/openshift-lws-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.451724177Z namespaces/openshift-lws-operator/coordination.k8s.io/leases/b8b2488c.x-k8s.io.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.451849419Z namespaces/openshift-lws-operator/coordination.k8s.io/leases/openshift-lws-operator-lock.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.451906041Z namespaces/openshift-lws-operator/core/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.451960422Z namespaces/openshift-lws-operator/core/configmaps.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.452049743Z namespaces/openshift-lws-operator/core/endpoints.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.452175606Z namespaces/openshift-lws-operator/core/events.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.452475122Z namespaces/openshift-lws-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.452572084Z namespaces/openshift-lws-operator/core/pods.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.452757588Z namespaces/openshift-lws-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.45287437Z namespaces/openshift-lws-operator/core/secrets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.453075504Z namespaces/openshift-lws-operator/core/services.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.453177816Z namespaces/openshift-lws-operator/discovery.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.453215527Z namespaces/openshift-lws-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.453250528Z namespaces/openshift-lws-operator/image.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.453305809Z namespaces/openshift-lws-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.45336614Z namespaces/openshift-lws-operator/k8s.ovn.org/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.453387331Z namespaces/openshift-lws-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.453501763Z namespaces/openshift-lws-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.453582375Z namespaces/openshift-lws-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.453632656Z namespaces/openshift-lws-operator/monitoring.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.453679757Z namespaces/openshift-lws-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.453737878Z namespaces/openshift-lws-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.453774588Z namespaces/openshift-lws-operator/monitoring.coreos.com/servicemonitors/lws-controller-manager-metrics-monitor.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.45383689Z namespaces/openshift-lws-operator/networking.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.45386155Z namespaces/openshift-lws-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.453923651Z namespaces/openshift-lws-operator/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.453934902Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.454007623Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.454325459Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.455535344Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.455644266Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/leader-worker-set.v1.0.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.455943163Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.456151217Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.456385361Z namespaces/openshift-lws-operator/operators.coreos.com/installplans/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.456407052Z namespaces/openshift-lws-operator/operators.coreos.com/installplans/install-488tz.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.456503044Z namespaces/openshift-lws-operator/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.456543464Z namespaces/openshift-lws-operator/operators.coreos.com/operatorconditions/leader-worker-set.v1.0.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.456596316Z namespaces/openshift-lws-operator/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.456644037Z namespaces/openshift-lws-operator/operators.coreos.com/operatorgroups/leader-worker-set.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.456688078Z namespaces/openshift-lws-operator/operators.coreos.com/subscriptions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.456734788Z namespaces/openshift-lws-operator/operators.coreos.com/subscriptions/leader-worker-set.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.45680432Z namespaces/openshift-lws-operator/pods/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.45681623Z namespaces/openshift-lws-operator/pods/lws-controller-manager-8994644bb-4xntv/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.456848481Z namespaces/openshift-lws-operator/pods/lws-controller-manager-8994644bb-4xntv/lws-controller-manager-8994644bb-4xntv.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.456934563Z namespaces/openshift-lws-operator/pods/lws-controller-manager-8994644bb-4xntv/manager/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.456945023Z namespaces/openshift-lws-operator/pods/lws-controller-manager-8994644bb-4xntv/manager/manager/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.456950553Z namespaces/openshift-lws-operator/pods/lws-controller-manager-8994644bb-4xntv/manager/manager/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.456990014Z namespaces/openshift-lws-operator/pods/lws-controller-manager-8994644bb-4xntv/manager/manager/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.457109986Z namespaces/openshift-lws-operator/pods/lws-controller-manager-8994644bb-4xntv/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.457210988Z namespaces/openshift-lws-operator/pods/lws-controller-manager-8994644bb-4xntv/manager/manager/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.457229989Z namespaces/openshift-lws-operator/pods/lws-controller-manager-8994644bb-j4rkm/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.457283459Z namespaces/openshift-lws-operator/pods/lws-controller-manager-8994644bb-j4rkm/lws-controller-manager-8994644bb-j4rkm.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.457359261Z namespaces/openshift-lws-operator/pods/lws-controller-manager-8994644bb-j4rkm/manager/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.457370031Z namespaces/openshift-lws-operator/pods/lws-controller-manager-8994644bb-j4rkm/manager/manager/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.457381501Z namespaces/openshift-lws-operator/pods/lws-controller-manager-8994644bb-j4rkm/manager/manager/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.457409742Z namespaces/openshift-lws-operator/pods/lws-controller-manager-8994644bb-j4rkm/manager/manager/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.457763539Z namespaces/openshift-lws-operator/pods/lws-controller-manager-8994644bb-j4rkm/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.457826141Z namespaces/openshift-lws-operator/pods/lws-controller-manager-8994644bb-j4rkm/manager/manager/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.457880232Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-9978k/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.457905942Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-9978k/openshift-lws-operator-fd8ccff4c-9978k.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.457987224Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-9978k/openshift-lws-operator/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.458004974Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-9978k/openshift-lws-operator/openshift-lws-operator/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.458012225Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-9978k/openshift-lws-operator/openshift-lws-operator/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.458044095Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-9978k/openshift-lws-operator/openshift-lws-operator/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.458238569Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-9978k/openshift-lws-operator/openshift-lws-operator/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.458333881Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-9978k/openshift-lws-operator/openshift-lws-operator/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.458362632Z namespaces/openshift-lws-operator/policy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.458407373Z namespaces/openshift-lws-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.458456143Z namespaces/openshift-lws-operator/route.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.458501464Z namespaces/openshift-lws-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.458542695Z namespaces/openshift-machine-api/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.458549835Z namespaces/openshift-machine-api/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.458554076Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.458598386Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.45877525Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.458967254Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.459108247Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.459435423Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.459623357Z namespaces/openshift-machine-config-operator/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.459632017Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.459636278Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.459654428Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/prometheusrules/machine-config-controller.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.45976828Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/prometheusrules/machine-config-daemon.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.459852612Z namespaces/openshift-machine-config-operator/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.459863492Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.459877232Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.460047816Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.46024111Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.460379202Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.460596437Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.460779551Z namespaces/openshift-marketplace/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.460789151Z namespaces/openshift-marketplace/monitoring.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.460793481Z namespaces/openshift-marketplace/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.460824711Z namespaces/openshift-marketplace/monitoring.coreos.com/prometheusrules/marketplace-alert-rules.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.460902683Z namespaces/openshift-marketplace/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.460915533Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.460963364Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/certified-operators.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.461063886Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/community-operators.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.461177519Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/redhat-marketplace.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.46126231Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/redhat-operators.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.461319861Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.461362622Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.461510956Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.461688539Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.461820802Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.462041716Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.462252871Z namespaces/openshift-monitoring/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.462291931Z namespaces/openshift-monitoring/openshift-monitoring.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.462365273Z namespaces/openshift-monitoring/apps.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.462477065Z namespaces/openshift-monitoring/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.462555647Z namespaces/openshift-monitoring/apps/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.462687839Z namespaces/openshift-monitoring/apps/daemonsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.462820282Z namespaces/openshift-monitoring/apps/deployments.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.463399654Z namespaces/openshift-monitoring/apps/replicasets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.463739191Z namespaces/openshift-monitoring/apps/statefulsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.463993616Z namespaces/openshift-monitoring/autoscaling/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.464015886Z namespaces/openshift-monitoring/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.464077848Z namespaces/openshift-monitoring/batch/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.464141549Z namespaces/openshift-monitoring/batch/cronjobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.464278572Z namespaces/openshift-monitoring/batch/jobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.464357903Z namespaces/openshift-monitoring/build.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.464388524Z namespaces/openshift-monitoring/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.464490926Z namespaces/openshift-monitoring/build.openshift.io/builds.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.464532877Z namespaces/openshift-monitoring/core/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.464605308Z namespaces/openshift-monitoring/core/configmaps.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.467709001Z namespaces/openshift-monitoring/core/endpoints.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.467852174Z namespaces/openshift-monitoring/core/events.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.468758003Z namespaces/openshift-monitoring/core/persistentvolumeclaims.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.468830724Z namespaces/openshift-monitoring/core/pods.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.469731162Z namespaces/openshift-monitoring/core/replicationcontrollers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.469869025Z namespaces/openshift-monitoring/core/secrets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.470442237Z namespaces/openshift-monitoring/core/services.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.47060666Z namespaces/openshift-monitoring/discovery.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.470650501Z namespaces/openshift-monitoring/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.470809404Z namespaces/openshift-monitoring/image.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.470828565Z namespaces/openshift-monitoring/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.470884546Z namespaces/openshift-monitoring/k8s.ovn.org/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.470937637Z namespaces/openshift-monitoring/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.471013258Z namespaces/openshift-monitoring/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.47108314Z namespaces/openshift-monitoring/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.471142711Z namespaces/openshift-monitoring/monitoring.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.471207292Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.471347765Z namespaces/openshift-monitoring/monitoring.coreos.com/alertmanagers/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.471374656Z namespaces/openshift-monitoring/monitoring.coreos.com/alertmanagers/main.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.471464538Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheuses/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.471482568Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheuses/k8s.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.47159111Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.471632211Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/alertmanager-main-rules.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.471742663Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/cluster-monitoring-operator-prometheus-rules.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.471923547Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/kube-state-metrics-rules.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.472010689Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/kubernetes-monitoring-rules.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.472319225Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/node-exporter-rules.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.472452827Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/prometheus-k8s-prometheus-rules.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.47257367Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/prometheus-k8s-thanos-sidecar-rules.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.472660222Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/prometheus-operator-rules.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.472780094Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/telemetry.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.472865416Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/thanos-querier.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.472937787Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.472981618Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/alertmanager-main.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.47307373Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/cluster-monitoring-operator.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.473184582Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/kube-state-metrics.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.473268054Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/kubelet.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.473371436Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/metrics-server.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.473450558Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/node-exporter.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.473620401Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/openshift-state-metrics.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.473717303Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/prometheus-k8s.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.473814225Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/prometheus-operator.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.473898287Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/telemeter-client.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.473988729Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/thanos-querier.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.474084031Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/thanos-sidecar.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.474173362Z namespaces/openshift-monitoring/networking.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.474224194Z namespaces/openshift-monitoring/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.474289615Z namespaces/openshift-monitoring/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.474305585Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.474317565Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.474501309Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.474784225Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.474963488Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.475187033Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.475440708Z namespaces/openshift-monitoring/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.475675353Z namespaces/openshift-monitoring/operators.coreos.com/operatorgroups/openshift-cluster-monitoring.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.475682703Z namespaces/openshift-monitoring/pods/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.475687103Z namespaces/openshift-monitoring/pods/alertmanager-main-0/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.475721984Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager-main-0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.475912638Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.475926678Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.475933568Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.475980559Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.476090951Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.476199783Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.476244944Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.476257215Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.476264795Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.476302426Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.476417858Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.476497589Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.476549821Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.476561931Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.476568111Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.476665563Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.476770045Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.476853897Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.476914658Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.476926768Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.476936658Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.476976649Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.477097892Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.477212094Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.477241155Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.477249895Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.477279585Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.477334086Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.477441669Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.47752255Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.477559221Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.477565431Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.477571711Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.477625943Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.477730705Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.477818286Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.477878288Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.477895098Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.477909348Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.477938549Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.478048181Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.478144953Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.478210284Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-gd67w/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.478264495Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-gd67w/cluster-monitoring-operator-548b489cb8-gd67w.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.478377368Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-gd67w/cluster-monitoring-operator/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.478386638Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-gd67w/cluster-monitoring-operator/cluster-monitoring-operator/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.478390838Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-gd67w/cluster-monitoring-operator/cluster-monitoring-operator/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.478431809Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-gd67w/cluster-monitoring-operator/cluster-monitoring-operator/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.479298176Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-gd67w/cluster-monitoring-operator/cluster-monitoring-operator/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.479376258Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-gd67w/cluster-monitoring-operator/cluster-monitoring-operator/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.479425619Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-vvvrp/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.47948306Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-vvvrp/kube-state-metrics-56d9f944b4-vvvrp.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.479604473Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-vvvrp/kube-rbac-proxy-main/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.479613133Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-vvvrp/kube-rbac-proxy-main/kube-rbac-proxy-main/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.479622613Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-vvvrp/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.479682434Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-vvvrp/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.479794427Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-vvvrp/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.479881708Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-vvvrp/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.479933079Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-vvvrp/kube-rbac-proxy-self/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.479943119Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-vvvrp/kube-rbac-proxy-self/kube-rbac-proxy-self/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.47994745Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-vvvrp/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.47998991Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-vvvrp/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.480113603Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-vvvrp/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.480286446Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-vvvrp/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.480319767Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-vvvrp/kube-state-metrics/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.480331007Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-vvvrp/kube-state-metrics/kube-state-metrics/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.480341918Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-vvvrp/kube-state-metrics/kube-state-metrics/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.480396209Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-vvvrp/kube-state-metrics/kube-state-metrics/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.480612463Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-vvvrp/kube-state-metrics/kube-state-metrics/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.480709195Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-vvvrp/kube-state-metrics/kube-state-metrics/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.480753626Z namespaces/openshift-monitoring/pods/metrics-server-64446b66cb-mc49d/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.480815717Z namespaces/openshift-monitoring/pods/metrics-server-64446b66cb-mc49d/metrics-server-64446b66cb-mc49d.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.480916049Z namespaces/openshift-monitoring/pods/metrics-server-64446b66cb-mc49d/metrics-server/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.48092451Z namespaces/openshift-monitoring/pods/metrics-server-64446b66cb-mc49d/metrics-server/metrics-server/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.48094349Z namespaces/openshift-monitoring/pods/metrics-server-64446b66cb-mc49d/metrics-server/metrics-server/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.480983221Z namespaces/openshift-monitoring/pods/metrics-server-64446b66cb-mc49d/metrics-server/metrics-server/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.481164614Z namespaces/openshift-monitoring/pods/metrics-server-64446b66cb-mc49d/metrics-server/metrics-server/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.481250356Z namespaces/openshift-monitoring/pods/metrics-server-64446b66cb-mc49d/metrics-server/metrics-server/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.481298207Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-llwkt/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.481351478Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-llwkt/monitoring-plugin-68b7fc9574-llwkt.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.4814463Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-llwkt/monitoring-plugin/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.48145786Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-llwkt/monitoring-plugin/monitoring-plugin/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.48147309Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-llwkt/monitoring-plugin/monitoring-plugin/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.481535212Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-llwkt/monitoring-plugin/monitoring-plugin/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.481641734Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-llwkt/monitoring-plugin/monitoring-plugin/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.481743816Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-llwkt/monitoring-plugin/monitoring-plugin/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.481784747Z namespaces/openshift-monitoring/pods/node-exporter-66zdn/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.481844348Z namespaces/openshift-monitoring/pods/node-exporter-66zdn/node-exporter-66zdn.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.481966131Z namespaces/openshift-monitoring/pods/node-exporter-66zdn/init-textfile/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.481977201Z namespaces/openshift-monitoring/pods/node-exporter-66zdn/init-textfile/init-textfile/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.481985561Z namespaces/openshift-monitoring/pods/node-exporter-66zdn/init-textfile/init-textfile/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.482043312Z namespaces/openshift-monitoring/pods/node-exporter-66zdn/init-textfile/init-textfile/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.482166664Z namespaces/openshift-monitoring/pods/node-exporter-66zdn/init-textfile/init-textfile/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.482257226Z namespaces/openshift-monitoring/pods/node-exporter-66zdn/init-textfile/init-textfile/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.482326518Z namespaces/openshift-monitoring/pods/node-exporter-66zdn/kube-rbac-proxy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.482339368Z namespaces/openshift-monitoring/pods/node-exporter-66zdn/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.482350368Z namespaces/openshift-monitoring/pods/node-exporter-66zdn/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.482399589Z namespaces/openshift-monitoring/pods/node-exporter-66zdn/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.482505002Z namespaces/openshift-monitoring/pods/node-exporter-66zdn/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.482609024Z namespaces/openshift-monitoring/pods/node-exporter-66zdn/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.482658194Z namespaces/openshift-monitoring/pods/node-exporter-66zdn/node-exporter/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.482667265Z namespaces/openshift-monitoring/pods/node-exporter-66zdn/node-exporter/node-exporter/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.482673875Z namespaces/openshift-monitoring/pods/node-exporter-66zdn/node-exporter/node-exporter/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.482753016Z namespaces/openshift-monitoring/pods/node-exporter-66zdn/node-exporter/node-exporter/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.482852809Z namespaces/openshift-monitoring/pods/node-exporter-66zdn/node-exporter/node-exporter/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.482954271Z namespaces/openshift-monitoring/pods/node-exporter-66zdn/node-exporter/node-exporter/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.483022232Z namespaces/openshift-monitoring/pods/node-exporter-n8mhj/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.483076513Z namespaces/openshift-monitoring/pods/node-exporter-n8mhj/node-exporter-n8mhj.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.483232726Z namespaces/openshift-monitoring/pods/node-exporter-n8mhj/init-textfile/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.483252217Z namespaces/openshift-monitoring/pods/node-exporter-n8mhj/init-textfile/init-textfile/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.483257927Z namespaces/openshift-monitoring/pods/node-exporter-n8mhj/init-textfile/init-textfile/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.483270697Z namespaces/openshift-monitoring/pods/node-exporter-n8mhj/init-textfile/init-textfile/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.483387039Z namespaces/openshift-monitoring/pods/node-exporter-n8mhj/init-textfile/init-textfile/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.483470081Z namespaces/openshift-monitoring/pods/node-exporter-n8mhj/init-textfile/init-textfile/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.483514922Z namespaces/openshift-monitoring/pods/node-exporter-n8mhj/kube-rbac-proxy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.483526862Z namespaces/openshift-monitoring/pods/node-exporter-n8mhj/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.483531232Z namespaces/openshift-monitoring/pods/node-exporter-n8mhj/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.483562473Z namespaces/openshift-monitoring/pods/node-exporter-n8mhj/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.483679135Z namespaces/openshift-monitoring/pods/node-exporter-n8mhj/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.483789037Z namespaces/openshift-monitoring/pods/node-exporter-n8mhj/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.483819368Z namespaces/openshift-monitoring/pods/node-exporter-n8mhj/node-exporter/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.483833938Z namespaces/openshift-monitoring/pods/node-exporter-n8mhj/node-exporter/node-exporter/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.483840549Z namespaces/openshift-monitoring/pods/node-exporter-n8mhj/node-exporter/node-exporter/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.48390108Z namespaces/openshift-monitoring/pods/node-exporter-n8mhj/node-exporter/node-exporter/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.484021242Z namespaces/openshift-monitoring/pods/node-exporter-n8mhj/node-exporter/node-exporter/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.484106414Z namespaces/openshift-monitoring/pods/node-exporter-n8mhj/node-exporter/node-exporter/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.484178005Z namespaces/openshift-monitoring/pods/node-exporter-qjjlj/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.484238577Z namespaces/openshift-monitoring/pods/node-exporter-qjjlj/node-exporter-qjjlj.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.484360749Z namespaces/openshift-monitoring/pods/node-exporter-qjjlj/init-textfile/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.48437392Z namespaces/openshift-monitoring/pods/node-exporter-qjjlj/init-textfile/init-textfile/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.484379969Z namespaces/openshift-monitoring/pods/node-exporter-qjjlj/init-textfile/init-textfile/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.48442121Z namespaces/openshift-monitoring/pods/node-exporter-qjjlj/init-textfile/init-textfile/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.484510252Z namespaces/openshift-monitoring/pods/node-exporter-qjjlj/init-textfile/init-textfile/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.484579333Z namespaces/openshift-monitoring/pods/node-exporter-qjjlj/init-textfile/init-textfile/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.484632335Z namespaces/openshift-monitoring/pods/node-exporter-qjjlj/kube-rbac-proxy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.484640385Z namespaces/openshift-monitoring/pods/node-exporter-qjjlj/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.484646895Z namespaces/openshift-monitoring/pods/node-exporter-qjjlj/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.484702246Z namespaces/openshift-monitoring/pods/node-exporter-qjjlj/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.484819649Z namespaces/openshift-monitoring/pods/node-exporter-qjjlj/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.48491946Z namespaces/openshift-monitoring/pods/node-exporter-qjjlj/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.484961161Z namespaces/openshift-monitoring/pods/node-exporter-qjjlj/node-exporter/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.484970532Z namespaces/openshift-monitoring/pods/node-exporter-qjjlj/node-exporter/node-exporter/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.484976892Z namespaces/openshift-monitoring/pods/node-exporter-qjjlj/node-exporter/node-exporter/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.485033373Z namespaces/openshift-monitoring/pods/node-exporter-qjjlj/node-exporter/node-exporter/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.485177026Z namespaces/openshift-monitoring/pods/node-exporter-qjjlj/node-exporter/node-exporter/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.485283558Z namespaces/openshift-monitoring/pods/node-exporter-qjjlj/node-exporter/node-exporter/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.485317808Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-4fth6/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.48539301Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-4fth6/openshift-state-metrics-b8f49f5c6-4fth6.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.485556513Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-4fth6/kube-rbac-proxy-main/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.485568044Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-4fth6/kube-rbac-proxy-main/kube-rbac-proxy-main/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.485577744Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-4fth6/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.485626465Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-4fth6/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.485748927Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-4fth6/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.485822849Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-4fth6/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.48587298Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-4fth6/kube-rbac-proxy-self/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.48588457Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-4fth6/kube-rbac-proxy-self/kube-rbac-proxy-self/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.48589563Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-4fth6/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.485963522Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-4fth6/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.486072914Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-4fth6/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.486201257Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-4fth6/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.486247638Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-4fth6/openshift-state-metrics/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.486257598Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-4fth6/openshift-state-metrics/openshift-state-metrics/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.486263688Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-4fth6/openshift-state-metrics/openshift-state-metrics/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.486313019Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-4fth6/openshift-state-metrics/openshift-state-metrics/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.486450051Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-4fth6/openshift-state-metrics/openshift-state-metrics/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.486537893Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-4fth6/openshift-state-metrics/openshift-state-metrics/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.486631005Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.486713587Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus-k8s-0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.486920691Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.486930611Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.486946002Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.486998183Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.487156266Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.487249598Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.487270078Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.487287099Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.487306129Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.48734682Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.487462862Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.487538264Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.487587285Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.487600165Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.487604405Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.487647406Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.487751398Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.48782759Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.48786651Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.48787458Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.48787842Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.487934012Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.488036014Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.488236188Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.488267698Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.488272679Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.488286769Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.48833441Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.488434672Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.488520893Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.488569085Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.488577365Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.488581215Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.488637396Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.489372141Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.489458573Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.489505173Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.489517194Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.489523024Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.489583855Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.489684317Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.489773549Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.48983147Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-ng7mq/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.489885561Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-ng7mq/prometheus-operator-84d69f794d-ng7mq.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.490020294Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-ng7mq/kube-rbac-proxy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.490034324Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-ng7mq/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.490040204Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-ng7mq/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.490066265Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-ng7mq/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.490231968Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-ng7mq/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.49033667Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-ng7mq/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.490353981Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-ng7mq/prometheus-operator/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.490365581Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-ng7mq/prometheus-operator/prometheus-operator/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.490372831Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-ng7mq/prometheus-operator/prometheus-operator/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.490445883Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-ng7mq/prometheus-operator/prometheus-operator/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.490733458Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-ng7mq/prometheus-operator/prometheus-operator/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.4908204Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-ng7mq/prometheus-operator/prometheus-operator/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.490870351Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-q6vnx/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.490928722Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-q6vnx/prometheus-operator-admission-webhook-7f8d78f54b-q6vnx.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.491079505Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-q6vnx/prometheus-operator-admission-webhook/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.491091506Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-q6vnx/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.491098146Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-q6vnx/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.491183218Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-q6vnx/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.49129008Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-q6vnx/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.491377122Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-q6vnx/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.491428253Z namespaces/openshift-monitoring/pods/telemeter-client-6f6697f644-mbplm/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.491499154Z namespaces/openshift-monitoring/pods/telemeter-client-6f6697f644-mbplm/telemeter-client-6f6697f644-mbplm.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.491647457Z namespaces/openshift-monitoring/pods/telemeter-client-6f6697f644-mbplm/kube-rbac-proxy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.491658697Z namespaces/openshift-monitoring/pods/telemeter-client-6f6697f644-mbplm/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.491662747Z namespaces/openshift-monitoring/pods/telemeter-client-6f6697f644-mbplm/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.491714428Z namespaces/openshift-monitoring/pods/telemeter-client-6f6697f644-mbplm/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.491844531Z namespaces/openshift-monitoring/pods/telemeter-client-6f6697f644-mbplm/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.491932443Z namespaces/openshift-monitoring/pods/telemeter-client-6f6697f644-mbplm/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.491970654Z namespaces/openshift-monitoring/pods/telemeter-client-6f6697f644-mbplm/reload/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.491980924Z namespaces/openshift-monitoring/pods/telemeter-client-6f6697f644-mbplm/reload/reload/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.491988264Z namespaces/openshift-monitoring/pods/telemeter-client-6f6697f644-mbplm/reload/reload/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.492048565Z namespaces/openshift-monitoring/pods/telemeter-client-6f6697f644-mbplm/reload/reload/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.492181978Z namespaces/openshift-monitoring/pods/telemeter-client-6f6697f644-mbplm/reload/reload/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.4922706Z namespaces/openshift-monitoring/pods/telemeter-client-6f6697f644-mbplm/reload/reload/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.492317581Z namespaces/openshift-monitoring/pods/telemeter-client-6f6697f644-mbplm/telemeter-client/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.492325701Z namespaces/openshift-monitoring/pods/telemeter-client-6f6697f644-mbplm/telemeter-client/telemeter-client/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.492329721Z namespaces/openshift-monitoring/pods/telemeter-client-6f6697f644-mbplm/telemeter-client/telemeter-client/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.492394832Z namespaces/openshift-monitoring/pods/telemeter-client-6f6697f644-mbplm/telemeter-client/telemeter-client/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.492512594Z namespaces/openshift-monitoring/pods/telemeter-client-6f6697f644-mbplm/telemeter-client/telemeter-client/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.492595816Z namespaces/openshift-monitoring/pods/telemeter-client-6f6697f644-mbplm/telemeter-client/telemeter-client/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.492644517Z namespaces/openshift-monitoring/pods/thanos-querier-75fc5bc9cc-clmpl/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.492738549Z namespaces/openshift-monitoring/pods/thanos-querier-75fc5bc9cc-clmpl/thanos-querier-75fc5bc9cc-clmpl.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.492891632Z namespaces/openshift-monitoring/pods/thanos-querier-75fc5bc9cc-clmpl/kube-rbac-proxy-metrics/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.492907062Z namespaces/openshift-monitoring/pods/thanos-querier-75fc5bc9cc-clmpl/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.492913983Z namespaces/openshift-monitoring/pods/thanos-querier-75fc5bc9cc-clmpl/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.492936693Z namespaces/openshift-monitoring/pods/thanos-querier-75fc5bc9cc-clmpl/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.493067036Z namespaces/openshift-monitoring/pods/thanos-querier-75fc5bc9cc-clmpl/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.493178238Z namespaces/openshift-monitoring/pods/thanos-querier-75fc5bc9cc-clmpl/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.49328936Z namespaces/openshift-monitoring/pods/thanos-querier-75fc5bc9cc-clmpl/kube-rbac-proxy-rules/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.493298861Z namespaces/openshift-monitoring/pods/thanos-querier-75fc5bc9cc-clmpl/kube-rbac-proxy-rules/kube-rbac-proxy-rules/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.493302861Z namespaces/openshift-monitoring/pods/thanos-querier-75fc5bc9cc-clmpl/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.493326461Z namespaces/openshift-monitoring/pods/thanos-querier-75fc5bc9cc-clmpl/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.493442223Z namespaces/openshift-monitoring/pods/thanos-querier-75fc5bc9cc-clmpl/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.493522905Z namespaces/openshift-monitoring/pods/thanos-querier-75fc5bc9cc-clmpl/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.493585026Z namespaces/openshift-monitoring/pods/thanos-querier-75fc5bc9cc-clmpl/kube-rbac-proxy-web/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.493599077Z namespaces/openshift-monitoring/pods/thanos-querier-75fc5bc9cc-clmpl/kube-rbac-proxy-web/kube-rbac-proxy-web/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.493604327Z namespaces/openshift-monitoring/pods/thanos-querier-75fc5bc9cc-clmpl/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.493648968Z namespaces/openshift-monitoring/pods/thanos-querier-75fc5bc9cc-clmpl/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.49375898Z namespaces/openshift-monitoring/pods/thanos-querier-75fc5bc9cc-clmpl/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.493840992Z namespaces/openshift-monitoring/pods/thanos-querier-75fc5bc9cc-clmpl/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.493887362Z namespaces/openshift-monitoring/pods/thanos-querier-75fc5bc9cc-clmpl/kube-rbac-proxy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.493897863Z namespaces/openshift-monitoring/pods/thanos-querier-75fc5bc9cc-clmpl/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.493905923Z namespaces/openshift-monitoring/pods/thanos-querier-75fc5bc9cc-clmpl/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.493949654Z namespaces/openshift-monitoring/pods/thanos-querier-75fc5bc9cc-clmpl/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.494065396Z namespaces/openshift-monitoring/pods/thanos-querier-75fc5bc9cc-clmpl/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.494179828Z namespaces/openshift-monitoring/pods/thanos-querier-75fc5bc9cc-clmpl/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.494202769Z namespaces/openshift-monitoring/pods/thanos-querier-75fc5bc9cc-clmpl/prom-label-proxy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.494207149Z namespaces/openshift-monitoring/pods/thanos-querier-75fc5bc9cc-clmpl/prom-label-proxy/prom-label-proxy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.494213129Z namespaces/openshift-monitoring/pods/thanos-querier-75fc5bc9cc-clmpl/prom-label-proxy/prom-label-proxy/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.49427477Z namespaces/openshift-monitoring/pods/thanos-querier-75fc5bc9cc-clmpl/prom-label-proxy/prom-label-proxy/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.494400283Z namespaces/openshift-monitoring/pods/thanos-querier-75fc5bc9cc-clmpl/prom-label-proxy/prom-label-proxy/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.494495145Z namespaces/openshift-monitoring/pods/thanos-querier-75fc5bc9cc-clmpl/prom-label-proxy/prom-label-proxy/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.494535246Z namespaces/openshift-monitoring/pods/thanos-querier-75fc5bc9cc-clmpl/thanos-query/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.494544006Z namespaces/openshift-monitoring/pods/thanos-querier-75fc5bc9cc-clmpl/thanos-query/thanos-query/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.494554016Z namespaces/openshift-monitoring/pods/thanos-querier-75fc5bc9cc-clmpl/thanos-query/thanos-query/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.494619567Z namespaces/openshift-monitoring/pods/thanos-querier-75fc5bc9cc-clmpl/thanos-query/thanos-query/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.49473342Z namespaces/openshift-monitoring/pods/thanos-querier-75fc5bc9cc-clmpl/thanos-query/thanos-query/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.494802701Z namespaces/openshift-monitoring/pods/thanos-querier-75fc5bc9cc-clmpl/thanos-query/thanos-query/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.494851462Z namespaces/openshift-monitoring/policy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.494902013Z namespaces/openshift-monitoring/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.494972804Z namespaces/openshift-monitoring/route.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.495002415Z namespaces/openshift-monitoring/route.openshift.io/routes.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.495148198Z namespaces/openshift-multus/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.495186579Z namespaces/openshift-multus/openshift-multus.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.49525801Z namespaces/openshift-multus/apps.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.495312481Z namespaces/openshift-multus/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.495368752Z namespaces/openshift-multus/apps/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.495414224Z namespaces/openshift-multus/apps/daemonsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.495659398Z namespaces/openshift-multus/apps/deployments.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.49574165Z namespaces/openshift-multus/apps/replicasets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.495838712Z namespaces/openshift-multus/apps/statefulsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.495896203Z namespaces/openshift-multus/apps/daemonsets/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.495944504Z namespaces/openshift-multus/apps/daemonsets/multus-additional-cni-plugins.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.496109687Z namespaces/openshift-multus/apps/daemonsets/multus.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.49625276Z namespaces/openshift-multus/apps/daemonsets/network-metrics-daemon.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.496332562Z namespaces/openshift-multus/autoscaling/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.496385133Z namespaces/openshift-multus/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.496444544Z namespaces/openshift-multus/batch/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.496482525Z namespaces/openshift-multus/batch/cronjobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.496587287Z namespaces/openshift-multus/batch/jobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.496635968Z namespaces/openshift-multus/build.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.496676979Z namespaces/openshift-multus/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.496793572Z namespaces/openshift-multus/build.openshift.io/builds.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.496846873Z namespaces/openshift-multus/core/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.496896034Z namespaces/openshift-multus/core/configmaps.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.497113388Z namespaces/openshift-multus/core/endpoints.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.497282971Z namespaces/openshift-multus/core/events.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.497818232Z namespaces/openshift-multus/core/persistentvolumeclaims.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.497975365Z namespaces/openshift-multus/core/pods.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.49868815Z namespaces/openshift-multus/core/replicationcontrollers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.498813082Z namespaces/openshift-multus/core/secrets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.498964825Z namespaces/openshift-multus/core/services.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.499037347Z namespaces/openshift-multus/core/configmaps/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.499090068Z namespaces/openshift-multus/core/configmaps/cni-copy-resources.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.499242091Z namespaces/openshift-multus/core/configmaps/default-cni-sysctl-allowlist.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.499362124Z namespaces/openshift-multus/core/configmaps/multus-daemon-config.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.499464066Z namespaces/openshift-multus/core/configmaps/whereabouts-flatfile-config.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.499583668Z namespaces/openshift-multus/core/serviceaccounts/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.499626949Z namespaces/openshift-multus/core/serviceaccounts/metrics-daemon-sa.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.499733731Z namespaces/openshift-multus/core/serviceaccounts/multus-ac.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.499848034Z namespaces/openshift-multus/core/serviceaccounts/multus-ancillary-tools.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.499979246Z namespaces/openshift-multus/core/serviceaccounts/multus.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.500048978Z namespaces/openshift-multus/core/services/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.50017754Z namespaces/openshift-multus/core/services/network-metrics-service.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.500237971Z namespaces/openshift-multus/discovery.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.500292003Z namespaces/openshift-multus/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.500362344Z namespaces/openshift-multus/image.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.500423575Z namespaces/openshift-multus/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.500489156Z namespaces/openshift-multus/k8s.ovn.org/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.500528467Z namespaces/openshift-multus/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.50064459Z namespaces/openshift-multus/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.500733531Z namespaces/openshift-multus/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.500808483Z namespaces/openshift-multus/monitoring.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.500835303Z namespaces/openshift-multus/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.500920445Z namespaces/openshift-multus/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.500965306Z namespaces/openshift-multus/monitoring.coreos.com/servicemonitors/monitor-network.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.501023517Z namespaces/openshift-multus/networking.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.501086628Z namespaces/openshift-multus/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.5011792Z namespaces/openshift-multus/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.501198851Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.501257762Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.501461226Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.50166118Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.501839074Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.502081119Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.502310864Z namespaces/openshift-multus/pods/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.502324694Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8xzdj/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.502381805Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8xzdj/multus-additional-cni-plugins-8xzdj.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.502543798Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8xzdj/bond-cni-plugin/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.502553498Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8xzdj/bond-cni-plugin/bond-cni-plugin/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.502572619Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8xzdj/bond-cni-plugin/bond-cni-plugin/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.502611879Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8xzdj/bond-cni-plugin/bond-cni-plugin/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.502731812Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8xzdj/bond-cni-plugin/bond-cni-plugin/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.502823854Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8xzdj/bond-cni-plugin/bond-cni-plugin/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.502878515Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8xzdj/cni-plugins/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.502889695Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8xzdj/cni-plugins/cni-plugins/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.502894095Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8xzdj/cni-plugins/cni-plugins/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.502938826Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8xzdj/cni-plugins/cni-plugins/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.503051768Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8xzdj/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.503164851Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8xzdj/cni-plugins/cni-plugins/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.503215072Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8xzdj/egress-router-binary-copy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.503226392Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8xzdj/egress-router-binary-copy/egress-router-binary-copy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.503231532Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8xzdj/egress-router-binary-copy/egress-router-binary-copy/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.503282883Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8xzdj/egress-router-binary-copy/egress-router-binary-copy/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.503391515Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8xzdj/egress-router-binary-copy/egress-router-binary-copy/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.503485247Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8xzdj/egress-router-binary-copy/egress-router-binary-copy/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.503513678Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8xzdj/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.503519278Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8xzdj/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.503584279Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8xzdj/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.50363369Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8xzdj/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.503731642Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8xzdj/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.503822504Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8xzdj/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.503839544Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8xzdj/routeoverride-cni/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.503854385Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8xzdj/routeoverride-cni/routeoverride-cni/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.503858735Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8xzdj/routeoverride-cni/routeoverride-cni/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.503919706Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8xzdj/routeoverride-cni/routeoverride-cni/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.504038239Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8xzdj/routeoverride-cni/routeoverride-cni/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.504149241Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8xzdj/routeoverride-cni/routeoverride-cni/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.504188102Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8xzdj/whereabouts-cni-bincopy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.504193182Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8xzdj/whereabouts-cni-bincopy/whereabouts-cni-bincopy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.504197542Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8xzdj/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.504248253Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8xzdj/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.504352225Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8xzdj/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.504442187Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8xzdj/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.504500638Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8xzdj/whereabouts-cni/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.504512398Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8xzdj/whereabouts-cni/whereabouts-cni/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.504516738Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8xzdj/whereabouts-cni/whereabouts-cni/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.504557569Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8xzdj/whereabouts-cni/whereabouts-cni/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.504663851Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8xzdj/whereabouts-cni/whereabouts-cni/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.504749843Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8xzdj/whereabouts-cni/whereabouts-cni/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.504795344Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lpb4n/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.504857915Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lpb4n/multus-additional-cni-plugins-lpb4n.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.505000978Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lpb4n/bond-cni-plugin/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.505011218Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lpb4n/bond-cni-plugin/bond-cni-plugin/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.505016298Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lpb4n/bond-cni-plugin/bond-cni-plugin/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.505040779Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lpb4n/bond-cni-plugin/bond-cni-plugin/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.505195762Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lpb4n/bond-cni-plugin/bond-cni-plugin/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.505276813Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lpb4n/bond-cni-plugin/bond-cni-plugin/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.505312594Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lpb4n/cni-plugins/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.505319565Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lpb4n/cni-plugins/cni-plugins/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.505324714Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lpb4n/cni-plugins/cni-plugins/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.505376686Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lpb4n/cni-plugins/cni-plugins/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.505481878Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lpb4n/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.505554099Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lpb4n/cni-plugins/cni-plugins/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.50559981Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lpb4n/egress-router-binary-copy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.50560926Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lpb4n/egress-router-binary-copy/egress-router-binary-copy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.50561872Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lpb4n/egress-router-binary-copy/egress-router-binary-copy/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.505653691Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lpb4n/egress-router-binary-copy/egress-router-binary-copy/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.505766203Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lpb4n/egress-router-binary-copy/egress-router-binary-copy/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.505834665Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lpb4n/egress-router-binary-copy/egress-router-binary-copy/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.505894396Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lpb4n/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.505906956Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lpb4n/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.505913226Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lpb4n/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.505938787Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lpb4n/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.506038509Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lpb4n/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.506148951Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lpb4n/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.506204492Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lpb4n/routeoverride-cni/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.506213152Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lpb4n/routeoverride-cni/routeoverride-cni/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.506217063Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lpb4n/routeoverride-cni/routeoverride-cni/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.506258384Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lpb4n/routeoverride-cni/routeoverride-cni/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.506369666Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lpb4n/routeoverride-cni/routeoverride-cni/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.506455877Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lpb4n/routeoverride-cni/routeoverride-cni/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.506504658Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lpb4n/whereabouts-cni-bincopy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.506513039Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lpb4n/whereabouts-cni-bincopy/whereabouts-cni-bincopy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.506517049Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lpb4n/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.5065803Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lpb4n/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.506669492Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lpb4n/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.506747614Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lpb4n/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.506798975Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lpb4n/whereabouts-cni/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.506807455Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lpb4n/whereabouts-cni/whereabouts-cni/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.506811315Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lpb4n/whereabouts-cni/whereabouts-cni/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.506870266Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lpb4n/whereabouts-cni/whereabouts-cni/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.506987028Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lpb4n/whereabouts-cni/whereabouts-cni/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.50708693Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lpb4n/whereabouts-cni/whereabouts-cni/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.507130651Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xnpnk/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.507300055Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xnpnk/multus-additional-cni-plugins-xnpnk.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.507436917Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xnpnk/bond-cni-plugin/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.507445038Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xnpnk/bond-cni-plugin/bond-cni-plugin/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.507451508Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xnpnk/bond-cni-plugin/bond-cni-plugin/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.507513559Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xnpnk/bond-cni-plugin/bond-cni-plugin/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.507635901Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xnpnk/bond-cni-plugin/bond-cni-plugin/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.507720663Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xnpnk/bond-cni-plugin/bond-cni-plugin/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.507766484Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xnpnk/cni-plugins/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.507775014Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xnpnk/cni-plugins/cni-plugins/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.507778714Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xnpnk/cni-plugins/cni-plugins/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.507831676Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xnpnk/cni-plugins/cni-plugins/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.507919957Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xnpnk/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.508018159Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xnpnk/cni-plugins/cni-plugins/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.5080648Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xnpnk/egress-router-binary-copy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.50807282Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xnpnk/egress-router-binary-copy/egress-router-binary-copy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.50807671Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xnpnk/egress-router-binary-copy/egress-router-binary-copy/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.508143512Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xnpnk/egress-router-binary-copy/egress-router-binary-copy/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.508242884Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xnpnk/egress-router-binary-copy/egress-router-binary-copy/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.508319085Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xnpnk/egress-router-binary-copy/egress-router-binary-copy/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.508358126Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xnpnk/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.508367416Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xnpnk/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.508372696Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xnpnk/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.508404887Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xnpnk/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.508514239Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xnpnk/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.508581301Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xnpnk/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.508624372Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xnpnk/routeoverride-cni/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.508632932Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xnpnk/routeoverride-cni/routeoverride-cni/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.508636772Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xnpnk/routeoverride-cni/routeoverride-cni/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.508681053Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xnpnk/routeoverride-cni/routeoverride-cni/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.508785655Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xnpnk/routeoverride-cni/routeoverride-cni/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.508863786Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xnpnk/routeoverride-cni/routeoverride-cni/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.508901527Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xnpnk/whereabouts-cni-bincopy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.508910257Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xnpnk/whereabouts-cni-bincopy/whereabouts-cni-bincopy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.508922918Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xnpnk/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.508978249Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xnpnk/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.509078261Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xnpnk/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.509177383Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xnpnk/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.509236904Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xnpnk/whereabouts-cni/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.509252934Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xnpnk/whereabouts-cni/whereabouts-cni/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.509258704Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xnpnk/whereabouts-cni/whereabouts-cni/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.509280365Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xnpnk/whereabouts-cni/whereabouts-cni/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.509388667Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xnpnk/whereabouts-cni/whereabouts-cni/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.509471909Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xnpnk/whereabouts-cni/whereabouts-cni/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.509508939Z namespaces/openshift-multus/pods/multus-f6vs6/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.509560831Z namespaces/openshift-multus/pods/multus-f6vs6/multus-f6vs6.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.509668113Z namespaces/openshift-multus/pods/multus-f6vs6/kube-multus/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.509676973Z namespaces/openshift-multus/pods/multus-f6vs6/kube-multus/kube-multus/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.509681063Z namespaces/openshift-multus/pods/multus-f6vs6/kube-multus/kube-multus/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.509708474Z namespaces/openshift-multus/pods/multus-f6vs6/kube-multus/kube-multus/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.510596441Z namespaces/openshift-multus/pods/multus-f6vs6/kube-multus/kube-multus/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.510670903Z namespaces/openshift-multus/pods/multus-f6vs6/kube-multus/kube-multus/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.510690203Z namespaces/openshift-multus/pods/multus-l4s6b/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.510778905Z namespaces/openshift-multus/pods/multus-l4s6b/multus-l4s6b.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.510887407Z namespaces/openshift-multus/pods/multus-l4s6b/kube-multus/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.510901528Z namespaces/openshift-multus/pods/multus-l4s6b/kube-multus/kube-multus/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.510907438Z namespaces/openshift-multus/pods/multus-l4s6b/kube-multus/kube-multus/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.510923538Z namespaces/openshift-multus/pods/multus-l4s6b/kube-multus/kube-multus/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.511766775Z namespaces/openshift-multus/pods/multus-l4s6b/kube-multus/kube-multus/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.511836707Z namespaces/openshift-multus/pods/multus-l4s6b/kube-multus/kube-multus/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.511886478Z namespaces/openshift-multus/pods/multus-lr8ct/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.511967529Z namespaces/openshift-multus/pods/multus-lr8ct/multus-lr8ct.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.512074912Z namespaces/openshift-multus/pods/multus-lr8ct/kube-multus/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.512087342Z namespaces/openshift-multus/pods/multus-lr8ct/kube-multus/kube-multus/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.512093542Z namespaces/openshift-multus/pods/multus-lr8ct/kube-multus/kube-multus/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.512153173Z namespaces/openshift-multus/pods/multus-lr8ct/kube-multus/kube-multus/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.512959769Z namespaces/openshift-multus/pods/multus-lr8ct/kube-multus/kube-multus/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.513026571Z namespaces/openshift-multus/pods/multus-lr8ct/kube-multus/kube-multus/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.513068972Z namespaces/openshift-multus/pods/network-metrics-daemon-77rmg/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.513107132Z namespaces/openshift-multus/pods/network-metrics-daemon-77rmg/network-metrics-daemon-77rmg.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.513220045Z namespaces/openshift-multus/pods/network-metrics-daemon-77rmg/kube-rbac-proxy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.513234555Z namespaces/openshift-multus/pods/network-metrics-daemon-77rmg/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.513250225Z namespaces/openshift-multus/pods/network-metrics-daemon-77rmg/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.513261036Z namespaces/openshift-multus/pods/network-metrics-daemon-77rmg/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.513396908Z namespaces/openshift-multus/pods/network-metrics-daemon-77rmg/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.51345902Z namespaces/openshift-multus/pods/network-metrics-daemon-77rmg/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.513508311Z namespaces/openshift-multus/pods/network-metrics-daemon-77rmg/network-metrics-daemon/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.513518541Z namespaces/openshift-multus/pods/network-metrics-daemon-77rmg/network-metrics-daemon/network-metrics-daemon/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.513523191Z namespaces/openshift-multus/pods/network-metrics-daemon-77rmg/network-metrics-daemon/network-metrics-daemon/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.513567252Z namespaces/openshift-multus/pods/network-metrics-daemon-77rmg/network-metrics-daemon/network-metrics-daemon/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.513750565Z namespaces/openshift-multus/pods/network-metrics-daemon-77rmg/network-metrics-daemon/network-metrics-daemon/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.513830987Z namespaces/openshift-multus/pods/network-metrics-daemon-77rmg/network-metrics-daemon/network-metrics-daemon/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.513877108Z namespaces/openshift-multus/pods/network-metrics-daemon-ng94v/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.513920499Z namespaces/openshift-multus/pods/network-metrics-daemon-ng94v/network-metrics-daemon-ng94v.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.514017061Z namespaces/openshift-multus/pods/network-metrics-daemon-ng94v/kube-rbac-proxy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.514028601Z namespaces/openshift-multus/pods/network-metrics-daemon-ng94v/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.514035541Z namespaces/openshift-multus/pods/network-metrics-daemon-ng94v/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.514081702Z namespaces/openshift-multus/pods/network-metrics-daemon-ng94v/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.514300347Z namespaces/openshift-multus/pods/network-metrics-daemon-ng94v/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.514392578Z namespaces/openshift-multus/pods/network-metrics-daemon-ng94v/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.514425899Z namespaces/openshift-multus/pods/network-metrics-daemon-ng94v/network-metrics-daemon/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.514435069Z namespaces/openshift-multus/pods/network-metrics-daemon-ng94v/network-metrics-daemon/network-metrics-daemon/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.51444663Z namespaces/openshift-multus/pods/network-metrics-daemon-ng94v/network-metrics-daemon/network-metrics-daemon/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.514516081Z namespaces/openshift-multus/pods/network-metrics-daemon-ng94v/network-metrics-daemon/network-metrics-daemon/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.514722575Z namespaces/openshift-multus/pods/network-metrics-daemon-ng94v/network-metrics-daemon/network-metrics-daemon/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.514816847Z namespaces/openshift-multus/pods/network-metrics-daemon-ng94v/network-metrics-daemon/network-metrics-daemon/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.514861808Z namespaces/openshift-multus/pods/network-metrics-daemon-zqjzm/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.514914279Z namespaces/openshift-multus/pods/network-metrics-daemon-zqjzm/network-metrics-daemon-zqjzm.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.515002981Z namespaces/openshift-multus/pods/network-metrics-daemon-zqjzm/kube-rbac-proxy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.515010871Z namespaces/openshift-multus/pods/network-metrics-daemon-zqjzm/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.515017451Z namespaces/openshift-multus/pods/network-metrics-daemon-zqjzm/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.515065182Z namespaces/openshift-multus/pods/network-metrics-daemon-zqjzm/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.515209085Z namespaces/openshift-multus/pods/network-metrics-daemon-zqjzm/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.515285587Z namespaces/openshift-multus/pods/network-metrics-daemon-zqjzm/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.515326768Z namespaces/openshift-multus/pods/network-metrics-daemon-zqjzm/network-metrics-daemon/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.515341818Z namespaces/openshift-multus/pods/network-metrics-daemon-zqjzm/network-metrics-daemon/network-metrics-daemon/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.515347308Z namespaces/openshift-multus/pods/network-metrics-daemon-zqjzm/network-metrics-daemon/network-metrics-daemon/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.515393749Z namespaces/openshift-multus/pods/network-metrics-daemon-zqjzm/network-metrics-daemon/network-metrics-daemon/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.515617934Z namespaces/openshift-multus/pods/network-metrics-daemon-zqjzm/network-metrics-daemon/network-metrics-daemon/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.515714565Z namespaces/openshift-multus/pods/network-metrics-daemon-zqjzm/network-metrics-daemon/network-metrics-daemon/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.515763796Z namespaces/openshift-multus/policy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.515801837Z namespaces/openshift-multus/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.515850218Z namespaces/openshift-multus/rbac.authorization.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.515857398Z namespaces/openshift-multus/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.515907829Z namespaces/openshift-multus/rbac.authorization.k8s.io/rolebindings/multus-whereabouts.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.516026902Z namespaces/openshift-multus/rbac.authorization.k8s.io/rolebindings/prometheus-k8s.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.516089593Z namespaces/openshift-multus/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.516140104Z namespaces/openshift-multus/rbac.authorization.k8s.io/roles/prometheus-k8s.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.516262047Z namespaces/openshift-multus/rbac.authorization.k8s.io/roles/whereabouts-cni.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.516330018Z namespaces/openshift-multus/route.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.516379189Z namespaces/openshift-multus/route.openshift.io/routes.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.51641649Z namespaces/openshift-must-gather-ct488/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.51642371Z namespaces/openshift-must-gather-ct488/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.51644104Z namespaces/openshift-must-gather-ct488/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.516485391Z namespaces/openshift-must-gather-ct488/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.516674165Z namespaces/openshift-must-gather-ct488/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.516875639Z namespaces/openshift-must-gather-ct488/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.517033502Z namespaces/openshift-must-gather-ct488/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.517283377Z namespaces/openshift-must-gather-ct488/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.517489661Z namespaces/openshift-network-console/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.517553943Z namespaces/openshift-network-console/openshift-network-console.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.517637884Z namespaces/openshift-network-console/apps.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.517671375Z namespaces/openshift-network-console/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.517751137Z namespaces/openshift-network-console/apps/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.517859629Z namespaces/openshift-network-console/apps/daemonsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.517956361Z namespaces/openshift-network-console/apps/deployments.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.518096394Z namespaces/openshift-network-console/apps/replicasets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.518315278Z namespaces/openshift-network-console/apps/statefulsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.518368429Z namespaces/openshift-network-console/apps/deployments/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.51841893Z namespaces/openshift-network-console/apps/deployments/networking-console-plugin.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.518506102Z namespaces/openshift-network-console/autoscaling/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.518582643Z namespaces/openshift-network-console/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.518651285Z namespaces/openshift-network-console/batch/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.518696866Z namespaces/openshift-network-console/batch/cronjobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.518795788Z namespaces/openshift-network-console/batch/jobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.518852919Z namespaces/openshift-network-console/build.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.51890582Z namespaces/openshift-network-console/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.519005722Z namespaces/openshift-network-console/build.openshift.io/builds.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.519047913Z namespaces/openshift-network-console/core/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.519104734Z namespaces/openshift-network-console/core/configmaps.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.519265077Z namespaces/openshift-network-console/core/endpoints.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.51937371Z namespaces/openshift-network-console/core/events.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.519551353Z namespaces/openshift-network-console/core/persistentvolumeclaims.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.519760698Z namespaces/openshift-network-console/core/pods.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.519947021Z namespaces/openshift-network-console/core/replicationcontrollers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.520090924Z namespaces/openshift-network-console/core/secrets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.520264648Z namespaces/openshift-network-console/core/services.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.520326229Z namespaces/openshift-network-console/core/configmaps/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.52038744Z namespaces/openshift-network-console/core/configmaps/networking-console-plugin.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.520458312Z namespaces/openshift-network-console/core/services/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.520517863Z namespaces/openshift-network-console/core/services/networking-console-plugin.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.520589774Z namespaces/openshift-network-console/discovery.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.520635155Z namespaces/openshift-network-console/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.520706757Z namespaces/openshift-network-console/image.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.520721617Z namespaces/openshift-network-console/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.520784768Z namespaces/openshift-network-console/k8s.ovn.org/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.520846209Z namespaces/openshift-network-console/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.521058474Z namespaces/openshift-network-console/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.521183406Z namespaces/openshift-network-console/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.521225577Z namespaces/openshift-network-console/monitoring.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.521280768Z namespaces/openshift-network-console/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.52133575Z namespaces/openshift-network-console/networking.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.521387491Z namespaces/openshift-network-console/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.521449652Z namespaces/openshift-network-console/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.521461622Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.521511183Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.521672576Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.52187196Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.522026664Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.522269608Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.522473962Z namespaces/openshift-network-console/pods/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.522481523Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-795tw/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.522527834Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-795tw/networking-console-plugin-74c9f884b7-795tw.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.522644186Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-795tw/networking-console-plugin/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.522652426Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-795tw/networking-console-plugin/networking-console-plugin/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.522656566Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-795tw/networking-console-plugin/networking-console-plugin/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.522735918Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-795tw/networking-console-plugin/networking-console-plugin/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.52284776Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-795tw/networking-console-plugin/networking-console-plugin/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.522933892Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-795tw/networking-console-plugin/networking-console-plugin/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.522970463Z namespaces/openshift-network-console/policy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.523025204Z namespaces/openshift-network-console/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.523064314Z namespaces/openshift-network-console/route.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.523141346Z namespaces/openshift-network-console/route.openshift.io/routes.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.523224148Z namespaces/openshift-network-diagnostics/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.523284439Z namespaces/openshift-network-diagnostics/openshift-network-diagnostics.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.523358351Z namespaces/openshift-network-diagnostics/apps.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.523413312Z namespaces/openshift-network-diagnostics/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.523469103Z namespaces/openshift-network-diagnostics/apps/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.523516464Z namespaces/openshift-network-diagnostics/apps/daemonsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.523654806Z namespaces/openshift-network-diagnostics/apps/deployments.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.523795549Z namespaces/openshift-network-diagnostics/apps/replicasets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.523913122Z namespaces/openshift-network-diagnostics/apps/statefulsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.523997444Z namespaces/openshift-network-diagnostics/apps/daemonsets/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.524051405Z namespaces/openshift-network-diagnostics/apps/daemonsets/network-check-target.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.524202178Z namespaces/openshift-network-diagnostics/apps/deployments/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.524229268Z namespaces/openshift-network-diagnostics/apps/deployments/network-check-source.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.52432294Z namespaces/openshift-network-diagnostics/autoscaling/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.524389512Z namespaces/openshift-network-diagnostics/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.524447573Z namespaces/openshift-network-diagnostics/batch/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.524501674Z namespaces/openshift-network-diagnostics/batch/cronjobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.524606536Z namespaces/openshift-network-diagnostics/batch/jobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.524665517Z namespaces/openshift-network-diagnostics/build.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.524722838Z namespaces/openshift-network-diagnostics/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.524871381Z namespaces/openshift-network-diagnostics/build.openshift.io/builds.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.524926542Z namespaces/openshift-network-diagnostics/core/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.524983673Z namespaces/openshift-network-diagnostics/core/configmaps.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.525103906Z namespaces/openshift-network-diagnostics/core/endpoints.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.525236379Z namespaces/openshift-network-diagnostics/core/events.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.525481614Z namespaces/openshift-network-diagnostics/core/persistentvolumeclaims.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.525561255Z namespaces/openshift-network-diagnostics/core/pods.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.525828851Z namespaces/openshift-network-diagnostics/core/replicationcontrollers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.525933773Z namespaces/openshift-network-diagnostics/core/secrets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.526063815Z namespaces/openshift-network-diagnostics/core/services.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.526208988Z namespaces/openshift-network-diagnostics/core/serviceaccounts/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.526265739Z namespaces/openshift-network-diagnostics/core/serviceaccounts/network-diagnostics.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.526332041Z namespaces/openshift-network-diagnostics/core/services/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.526387142Z namespaces/openshift-network-diagnostics/core/services/network-check-source.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.526495504Z namespaces/openshift-network-diagnostics/core/services/network-check-target.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.526555315Z namespaces/openshift-network-diagnostics/discovery.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.526615486Z namespaces/openshift-network-diagnostics/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.526688948Z namespaces/openshift-network-diagnostics/image.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.526724229Z namespaces/openshift-network-diagnostics/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.52678081Z namespaces/openshift-network-diagnostics/k8s.ovn.org/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.526835641Z namespaces/openshift-network-diagnostics/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.526938273Z namespaces/openshift-network-diagnostics/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.527048855Z namespaces/openshift-network-diagnostics/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.527096646Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.527137017Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.527222339Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.52727807Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/servicemonitors/network-check-source.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.527341351Z namespaces/openshift-network-diagnostics/networking.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.527390062Z namespaces/openshift-network-diagnostics/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.527441433Z namespaces/openshift-network-diagnostics/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.527449894Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.527497275Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.527697008Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.528054606Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.52823671Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.528473984Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.528673928Z namespaces/openshift-network-diagnostics/pods/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.528682679Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-mh7dr/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.52873066Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-mh7dr/network-check-source-7d88cb55d8-mh7dr.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.528827671Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-mh7dr/check-endpoints/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.528840432Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-mh7dr/check-endpoints/check-endpoints/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.528844622Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-mh7dr/check-endpoints/check-endpoints/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.528877623Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-mh7dr/check-endpoints/check-endpoints/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.528998445Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-mh7dr/check-endpoints/check-endpoints/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.529087097Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-mh7dr/check-endpoints/check-endpoints/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.529147698Z namespaces/openshift-network-diagnostics/pods/network-check-target-4gsqc/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.529210829Z namespaces/openshift-network-diagnostics/pods/network-check-target-4gsqc/network-check-target-4gsqc.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.529289581Z namespaces/openshift-network-diagnostics/pods/network-check-target-4gsqc/network-check-target-container/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.529298521Z namespaces/openshift-network-diagnostics/pods/network-check-target-4gsqc/network-check-target-container/network-check-target-container/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.529302591Z namespaces/openshift-network-diagnostics/pods/network-check-target-4gsqc/network-check-target-container/network-check-target-container/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.529348392Z namespaces/openshift-network-diagnostics/pods/network-check-target-4gsqc/network-check-target-container/network-check-target-container/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.529461954Z namespaces/openshift-network-diagnostics/pods/network-check-target-4gsqc/network-check-target-container/network-check-target-container/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.529538626Z namespaces/openshift-network-diagnostics/pods/network-check-target-4gsqc/network-check-target-container/network-check-target-container/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.529579967Z namespaces/openshift-network-diagnostics/pods/network-check-target-n6cbg/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.529633518Z namespaces/openshift-network-diagnostics/pods/network-check-target-n6cbg/network-check-target-n6cbg.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.52971756Z namespaces/openshift-network-diagnostics/pods/network-check-target-n6cbg/network-check-target-container/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.52973025Z namespaces/openshift-network-diagnostics/pods/network-check-target-n6cbg/network-check-target-container/network-check-target-container/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.52973654Z namespaces/openshift-network-diagnostics/pods/network-check-target-n6cbg/network-check-target-container/network-check-target-container/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.529773631Z namespaces/openshift-network-diagnostics/pods/network-check-target-n6cbg/network-check-target-container/network-check-target-container/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.529877713Z namespaces/openshift-network-diagnostics/pods/network-check-target-n6cbg/network-check-target-container/network-check-target-container/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.529970635Z namespaces/openshift-network-diagnostics/pods/network-check-target-n6cbg/network-check-target-container/network-check-target-container/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.530019926Z namespaces/openshift-network-diagnostics/pods/network-check-target-xrgmh/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.530082757Z namespaces/openshift-network-diagnostics/pods/network-check-target-xrgmh/network-check-target-xrgmh.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.530172479Z namespaces/openshift-network-diagnostics/pods/network-check-target-xrgmh/network-check-target-container/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.530188289Z namespaces/openshift-network-diagnostics/pods/network-check-target-xrgmh/network-check-target-container/network-check-target-container/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.530193569Z namespaces/openshift-network-diagnostics/pods/network-check-target-xrgmh/network-check-target-container/network-check-target-container/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.53023259Z namespaces/openshift-network-diagnostics/pods/network-check-target-xrgmh/network-check-target-container/network-check-target-container/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.530345642Z namespaces/openshift-network-diagnostics/pods/network-check-target-xrgmh/network-check-target-container/network-check-target-container/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.530437634Z namespaces/openshift-network-diagnostics/pods/network-check-target-xrgmh/network-check-target-container/network-check-target-container/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.530480735Z namespaces/openshift-network-diagnostics/policy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.530540616Z namespaces/openshift-network-diagnostics/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.530612998Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.530622998Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.530651888Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/rolebindings/network-diagnostics.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.530773811Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/rolebindings/prometheus-k8s.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.530848652Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.530905754Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/roles/network-diagnostics.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.531003726Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/roles/prometheus-k8s.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.531076137Z namespaces/openshift-network-diagnostics/route.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.531148949Z namespaces/openshift-network-diagnostics/route.openshift.io/routes.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.53121387Z namespaces/openshift-network-node-identity/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.531268021Z namespaces/openshift-network-node-identity/openshift-network-node-identity.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.531357113Z namespaces/openshift-network-node-identity/apps.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.531414944Z namespaces/openshift-network-node-identity/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.531475565Z namespaces/openshift-network-node-identity/apps/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.531522186Z namespaces/openshift-network-node-identity/apps/daemonsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.531619388Z namespaces/openshift-network-node-identity/apps/deployments.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.53171276Z namespaces/openshift-network-node-identity/apps/replicasets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.531849763Z namespaces/openshift-network-node-identity/apps/statefulsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.531876613Z namespaces/openshift-network-node-identity/autoscaling/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.531934165Z namespaces/openshift-network-node-identity/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.531983115Z namespaces/openshift-network-node-identity/batch/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.532037917Z namespaces/openshift-network-node-identity/batch/cronjobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.532138639Z namespaces/openshift-network-node-identity/batch/jobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.53221223Z namespaces/openshift-network-node-identity/build.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.532257381Z namespaces/openshift-network-node-identity/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.532357003Z namespaces/openshift-network-node-identity/build.openshift.io/builds.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.532409644Z namespaces/openshift-network-node-identity/coordination.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.532418144Z namespaces/openshift-network-node-identity/coordination.k8s.io/leases/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.532469665Z namespaces/openshift-network-node-identity/coordination.k8s.io/leases/ovnkube-identity.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.532535277Z namespaces/openshift-network-node-identity/core/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.532582228Z namespaces/openshift-network-node-identity/core/configmaps.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.53270489Z namespaces/openshift-network-node-identity/core/endpoints.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.532814172Z namespaces/openshift-network-node-identity/core/events.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.532900874Z namespaces/openshift-network-node-identity/core/persistentvolumeclaims.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.533004646Z namespaces/openshift-network-node-identity/core/pods.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.533094888Z namespaces/openshift-network-node-identity/core/replicationcontrollers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.533308222Z namespaces/openshift-network-node-identity/core/secrets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.533433905Z namespaces/openshift-network-node-identity/core/services.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.533480586Z namespaces/openshift-network-node-identity/core/serviceaccounts/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.533515157Z namespaces/openshift-network-node-identity/core/serviceaccounts/network-node-identity.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.533585028Z namespaces/openshift-network-node-identity/discovery.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.533629799Z namespaces/openshift-network-node-identity/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.53369376Z namespaces/openshift-network-node-identity/image.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.533749741Z namespaces/openshift-network-node-identity/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.533793342Z namespaces/openshift-network-node-identity/k8s.ovn.org/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.533848423Z namespaces/openshift-network-node-identity/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.533955866Z namespaces/openshift-network-node-identity/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.534035157Z namespaces/openshift-network-node-identity/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.534095438Z namespaces/openshift-network-node-identity/monitoring.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.5341701Z namespaces/openshift-network-node-identity/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.534214531Z namespaces/openshift-network-node-identity/networking.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.534276192Z namespaces/openshift-network-node-identity/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.534338313Z namespaces/openshift-network-node-identity/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.534352994Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.534398755Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.534584668Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.534780602Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.534930245Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.535224141Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.535432145Z namespaces/openshift-network-node-identity/policy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.535462186Z namespaces/openshift-network-node-identity/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.535532918Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.535542058Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.535594189Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/rolebindings/network-node-identity-leases.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.5356575Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.535711201Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/roles/network-node-identity-leases.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.535789523Z namespaces/openshift-network-node-identity/route.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.535828524Z namespaces/openshift-network-node-identity/route.openshift.io/routes.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.535896255Z namespaces/openshift-network-operator/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.535953646Z namespaces/openshift-network-operator/openshift-network-operator.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.536198151Z namespaces/openshift-network-operator/apps.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.536219272Z namespaces/openshift-network-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.536292513Z namespaces/openshift-network-operator/apps/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.536340864Z namespaces/openshift-network-operator/apps/daemonsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.536475727Z namespaces/openshift-network-operator/apps/deployments.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.536556608Z namespaces/openshift-network-operator/apps/replicasets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.536668261Z namespaces/openshift-network-operator/apps/statefulsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.536722962Z namespaces/openshift-network-operator/apps/daemonsets/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.536776773Z namespaces/openshift-network-operator/apps/daemonsets/iptables-alerter.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.536880045Z namespaces/openshift-network-operator/autoscaling/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.536925426Z namespaces/openshift-network-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.536990377Z namespaces/openshift-network-operator/batch/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.537026818Z namespaces/openshift-network-operator/batch/cronjobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.53715111Z namespaces/openshift-network-operator/batch/jobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.537224942Z namespaces/openshift-network-operator/build.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.537280313Z namespaces/openshift-network-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.537372625Z namespaces/openshift-network-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.537442896Z namespaces/openshift-network-operator/coordination.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.537455326Z namespaces/openshift-network-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.537495387Z namespaces/openshift-network-operator/coordination.k8s.io/leases/network-operator-lock.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.537565749Z namespaces/openshift-network-operator/core/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.53762105Z namespaces/openshift-network-operator/core/configmaps.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.537844405Z namespaces/openshift-network-operator/core/endpoints.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.537947727Z namespaces/openshift-network-operator/core/events.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.538156041Z namespaces/openshift-network-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.538254163Z namespaces/openshift-network-operator/core/pods.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.538458957Z namespaces/openshift-network-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.538554429Z namespaces/openshift-network-operator/core/secrets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.538740093Z namespaces/openshift-network-operator/core/services.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.538799674Z namespaces/openshift-network-operator/core/configmaps/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.538856305Z namespaces/openshift-network-operator/core/configmaps/applied-cluster.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.538961977Z namespaces/openshift-network-operator/core/configmaps/iptables-alerter-script.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.539032969Z namespaces/openshift-network-operator/core/serviceaccounts/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.53908316Z namespaces/openshift-network-operator/core/serviceaccounts/iptables-alerter.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.539165831Z namespaces/openshift-network-operator/discovery.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.539232093Z namespaces/openshift-network-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.539289294Z namespaces/openshift-network-operator/image.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.539334965Z namespaces/openshift-network-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.539379476Z namespaces/openshift-network-operator/k8s.ovn.org/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.539439347Z namespaces/openshift-network-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.539549559Z namespaces/openshift-network-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.539634541Z namespaces/openshift-network-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.539696752Z namespaces/openshift-network-operator/monitoring.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.539750463Z namespaces/openshift-network-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.539824895Z namespaces/openshift-network-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.539861005Z namespaces/openshift-network-operator/monitoring.coreos.com/prometheusrules/openshift-network-operator-ipsec-rules.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.539945497Z namespaces/openshift-network-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.539983058Z namespaces/openshift-network-operator/monitoring.coreos.com/servicemonitors/network-operator.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.540043589Z namespaces/openshift-network-operator/networking.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.54008329Z namespaces/openshift-network-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.540169272Z namespaces/openshift-network-operator/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.540186032Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.540243983Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.540426537Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.540623631Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.540782684Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.541010109Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.541251004Z namespaces/openshift-network-operator/pods/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.541262964Z namespaces/openshift-network-operator/pods/iptables-alerter-7hs9z/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.541299975Z namespaces/openshift-network-operator/pods/iptables-alerter-7hs9z/iptables-alerter-7hs9z.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.541397417Z namespaces/openshift-network-operator/pods/iptables-alerter-7hs9z/iptables-alerter/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.541407917Z namespaces/openshift-network-operator/pods/iptables-alerter-7hs9z/iptables-alerter/iptables-alerter/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.541411957Z namespaces/openshift-network-operator/pods/iptables-alerter-7hs9z/iptables-alerter/iptables-alerter/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.541460208Z namespaces/openshift-network-operator/pods/iptables-alerter-7hs9z/iptables-alerter/iptables-alerter/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.54156795Z namespaces/openshift-network-operator/pods/iptables-alerter-7hs9z/iptables-alerter/iptables-alerter/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.541663282Z namespaces/openshift-network-operator/pods/iptables-alerter-7hs9z/iptables-alerter/iptables-alerter/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.541706063Z namespaces/openshift-network-operator/pods/iptables-alerter-gqqgn/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.541758444Z namespaces/openshift-network-operator/pods/iptables-alerter-gqqgn/iptables-alerter-gqqgn.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.541835286Z namespaces/openshift-network-operator/pods/iptables-alerter-gqqgn/iptables-alerter/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.541843206Z namespaces/openshift-network-operator/pods/iptables-alerter-gqqgn/iptables-alerter/iptables-alerter/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.541847116Z namespaces/openshift-network-operator/pods/iptables-alerter-gqqgn/iptables-alerter/iptables-alerter/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.541888267Z namespaces/openshift-network-operator/pods/iptables-alerter-gqqgn/iptables-alerter/iptables-alerter/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.542001289Z namespaces/openshift-network-operator/pods/iptables-alerter-gqqgn/iptables-alerter/iptables-alerter/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.542114101Z namespaces/openshift-network-operator/pods/iptables-alerter-gqqgn/iptables-alerter/iptables-alerter/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.542178913Z namespaces/openshift-network-operator/pods/iptables-alerter-m4x6s/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.542231574Z namespaces/openshift-network-operator/pods/iptables-alerter-m4x6s/iptables-alerter-m4x6s.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.542316325Z namespaces/openshift-network-operator/pods/iptables-alerter-m4x6s/iptables-alerter/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.542324255Z namespaces/openshift-network-operator/pods/iptables-alerter-m4x6s/iptables-alerter/iptables-alerter/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.542328065Z namespaces/openshift-network-operator/pods/iptables-alerter-m4x6s/iptables-alerter/iptables-alerter/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.542370806Z namespaces/openshift-network-operator/pods/iptables-alerter-m4x6s/iptables-alerter/iptables-alerter/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.542481539Z namespaces/openshift-network-operator/pods/iptables-alerter-m4x6s/iptables-alerter/iptables-alerter/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.54257475Z namespaces/openshift-network-operator/pods/iptables-alerter-m4x6s/iptables-alerter/iptables-alerter/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.542626782Z namespaces/openshift-network-operator/policy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.542681503Z namespaces/openshift-network-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.542740894Z namespaces/openshift-network-operator/route.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.542806155Z namespaces/openshift-network-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.542848146Z namespaces/openshift-node/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.542863076Z namespaces/openshift-node/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.542869187Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.542914888Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.543096741Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.543343806Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.543483699Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.543731974Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.543955349Z namespaces/openshift-operator-lifecycle-manager/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.54403798Z namespaces/openshift-operator-lifecycle-manager/openshift-operator-lifecycle-manager.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.544115232Z namespaces/openshift-operator-lifecycle-manager/apps.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.544254884Z namespaces/openshift-operator-lifecycle-manager/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.544300025Z namespaces/openshift-operator-lifecycle-manager/apps/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.544359817Z namespaces/openshift-operator-lifecycle-manager/apps/daemonsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.544437258Z namespaces/openshift-operator-lifecycle-manager/apps/deployments.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.5445193Z namespaces/openshift-operator-lifecycle-manager/apps/replicasets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.544600971Z namespaces/openshift-operator-lifecycle-manager/apps/statefulsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.544657693Z namespaces/openshift-operator-lifecycle-manager/autoscaling/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.544682903Z namespaces/openshift-operator-lifecycle-manager/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.544732674Z namespaces/openshift-operator-lifecycle-manager/batch/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.544777525Z namespaces/openshift-operator-lifecycle-manager/batch/cronjobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.544855207Z namespaces/openshift-operator-lifecycle-manager/batch/jobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.544885037Z namespaces/openshift-operator-lifecycle-manager/build.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.544972679Z namespaces/openshift-operator-lifecycle-manager/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.545056641Z namespaces/openshift-operator-lifecycle-manager/build.openshift.io/builds.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.545106192Z namespaces/openshift-operator-lifecycle-manager/core/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.545185214Z namespaces/openshift-operator-lifecycle-manager/core/configmaps.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.545335486Z namespaces/openshift-operator-lifecycle-manager/core/endpoints.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.545443239Z namespaces/openshift-operator-lifecycle-manager/core/events.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.545536541Z namespaces/openshift-operator-lifecycle-manager/core/persistentvolumeclaims.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.545613072Z namespaces/openshift-operator-lifecycle-manager/core/pods.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.545688804Z namespaces/openshift-operator-lifecycle-manager/core/replicationcontrollers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.545826567Z namespaces/openshift-operator-lifecycle-manager/core/secrets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.546071701Z namespaces/openshift-operator-lifecycle-manager/core/services.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.546157443Z namespaces/openshift-operator-lifecycle-manager/discovery.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.546219904Z namespaces/openshift-operator-lifecycle-manager/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.546292696Z namespaces/openshift-operator-lifecycle-manager/image.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.546329757Z namespaces/openshift-operator-lifecycle-manager/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.546400508Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.546460649Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.546546421Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.546673954Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.546720765Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.546777346Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.546833437Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.546898998Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/prometheusrules/olm-alert-rules.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.54696707Z namespaces/openshift-operator-lifecycle-manager/networking.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.547016561Z namespaces/openshift-operator-lifecycle-manager/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.547115173Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.547148843Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.547182234Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.547354898Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.547553991Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.547723565Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.548192034Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.548399129Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.548422949Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/operatorgroups/olm-operators.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.54849333Z namespaces/openshift-operator-lifecycle-manager/policy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.548541311Z namespaces/openshift-operator-lifecycle-manager/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.548599743Z namespaces/openshift-operator-lifecycle-manager/route.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.548635473Z namespaces/openshift-operator-lifecycle-manager/route.openshift.io/routes.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.548675824Z namespaces/openshift-operators/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.548684494Z namespaces/openshift-operators/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.548689005Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.548732255Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.548881338Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.549048152Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.549211415Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.549407019Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.549601363Z namespaces/openshift-operators/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.549618453Z namespaces/openshift-operators/operators.coreos.com/operatorgroups/global-operators.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.549671745Z namespaces/openshift-ovn-kubernetes/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.549731246Z namespaces/openshift-ovn-kubernetes/openshift-ovn-kubernetes.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.549785417Z namespaces/openshift-ovn-kubernetes/apps.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.549824388Z namespaces/openshift-ovn-kubernetes/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.549870429Z namespaces/openshift-ovn-kubernetes/apps/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.549900029Z namespaces/openshift-ovn-kubernetes/apps/daemonsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.550083803Z namespaces/openshift-ovn-kubernetes/apps/deployments.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.550181915Z namespaces/openshift-ovn-kubernetes/apps/replicasets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.550263417Z namespaces/openshift-ovn-kubernetes/apps/statefulsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.550309418Z namespaces/openshift-ovn-kubernetes/apps/daemonsets/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.550348568Z namespaces/openshift-ovn-kubernetes/apps/daemonsets/ovnkube-node.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.550508322Z namespaces/openshift-ovn-kubernetes/autoscaling/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.550542572Z namespaces/openshift-ovn-kubernetes/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.550602123Z namespaces/openshift-ovn-kubernetes/batch/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.550633494Z namespaces/openshift-ovn-kubernetes/batch/cronjobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.550720826Z namespaces/openshift-ovn-kubernetes/batch/jobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.550752007Z namespaces/openshift-ovn-kubernetes/build.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.550797717Z namespaces/openshift-ovn-kubernetes/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.550878769Z namespaces/openshift-ovn-kubernetes/build.openshift.io/builds.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.55092468Z namespaces/openshift-ovn-kubernetes/coordination.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.55093309Z namespaces/openshift-ovn-kubernetes/coordination.k8s.io/leases/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.550966241Z namespaces/openshift-ovn-kubernetes/coordination.k8s.io/leases/ovn-kubernetes-master.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.551013812Z namespaces/openshift-ovn-kubernetes/core/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.551051693Z namespaces/openshift-ovn-kubernetes/core/configmaps.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.551304048Z namespaces/openshift-ovn-kubernetes/core/endpoints.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.551450401Z namespaces/openshift-ovn-kubernetes/core/events.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.551786648Z namespaces/openshift-ovn-kubernetes/core/persistentvolumeclaims.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.551935331Z namespaces/openshift-ovn-kubernetes/core/pods.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.552655075Z namespaces/openshift-ovn-kubernetes/core/replicationcontrollers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.552747297Z namespaces/openshift-ovn-kubernetes/core/secrets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.552942541Z namespaces/openshift-ovn-kubernetes/core/services.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.553001112Z namespaces/openshift-ovn-kubernetes/core/configmaps/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.553059613Z namespaces/openshift-ovn-kubernetes/core/configmaps/ovnkube-config.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.553203476Z namespaces/openshift-ovn-kubernetes/core/configmaps/ovnkube-script-lib.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.553357069Z namespaces/openshift-ovn-kubernetes/core/serviceaccounts/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.55340286Z namespaces/openshift-ovn-kubernetes/core/serviceaccounts/ovn-kubernetes-control-plane.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.553516543Z namespaces/openshift-ovn-kubernetes/core/serviceaccounts/ovn-kubernetes-node.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.553568213Z namespaces/openshift-ovn-kubernetes/core/services/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.553628535Z namespaces/openshift-ovn-kubernetes/core/services/ovn-kubernetes-node.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.553685946Z namespaces/openshift-ovn-kubernetes/discovery.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.553727877Z namespaces/openshift-ovn-kubernetes/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.553810869Z namespaces/openshift-ovn-kubernetes/image.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.553861539Z namespaces/openshift-ovn-kubernetes/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.553920891Z namespaces/openshift-ovn-kubernetes/k8s.cni.cncf.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.553933971Z namespaces/openshift-ovn-kubernetes/k8s.cni.cncf.io/network-attachment-definitions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.553978252Z namespaces/openshift-ovn-kubernetes/k8s.cni.cncf.io/network-attachment-definitions/default.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.554046203Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.554101084Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.554221857Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.554306799Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.55436849Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.554423801Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.554490472Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.554549733Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/prometheusrules/networking-rules.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.554656996Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.554742097Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/servicemonitors/monitor-ovn-node.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.554809689Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.554818089Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/operatorpkis/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.55486535Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/operatorpkis/ovn.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.554979762Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/operatorpkis/signer.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.555038993Z namespaces/openshift-ovn-kubernetes/networking.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.555100265Z namespaces/openshift-ovn-kubernetes/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.555175016Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.555193217Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.555234517Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.555407211Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.555604525Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.555760518Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.555990433Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.556228348Z namespaces/openshift-ovn-kubernetes/pods/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.556239668Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c954x/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.556313019Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c954x/ovnkube-node-c954x.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.556502213Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c954x/kube-rbac-proxy-node/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.556510203Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c954x/kube-rbac-proxy-node/kube-rbac-proxy-node/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.556514504Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c954x/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.556565105Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c954x/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.556709357Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c954x/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.556787829Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c954x/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.55684302Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c954x/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.55685436Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c954x/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.55686047Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c954x/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.556892221Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c954x/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.557041274Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c954x/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.557151206Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c954x/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.557196797Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c954x/nbdb/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.557208687Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c954x/nbdb/nbdb/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.557212907Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c954x/nbdb/nbdb/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.557253228Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c954x/nbdb/nbdb/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.557371191Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c954x/nbdb/nbdb/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.557477463Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c954x/nbdb/nbdb/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.557527834Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c954x/northd/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.557536894Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c954x/northd/northd/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.557541274Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c954x/northd/northd/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.557581475Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c954x/northd/northd/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.557710378Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c954x/northd/northd/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.557789679Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c954x/northd/northd/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.55783455Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c954x/ovn-acl-logging/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.55784543Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c954x/ovn-acl-logging/ovn-acl-logging/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.5578497Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c954x/ovn-acl-logging/ovn-acl-logging/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.557897201Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c954x/ovn-acl-logging/ovn-acl-logging/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.558006314Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c954x/ovn-acl-logging/ovn-acl-logging/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.558099625Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c954x/ovn-acl-logging/ovn-acl-logging/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.558154206Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c954x/ovn-controller/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.558177857Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c954x/ovn-controller/ovn-controller/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.558188857Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c954x/ovn-controller/ovn-controller/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.558206698Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c954x/ovn-controller/ovn-controller/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.558670437Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c954x/ovn-controller/ovn-controller/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.558763039Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c954x/ovn-controller/ovn-controller/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.55881049Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c954x/ovnkube-controller/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.5588183Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c954x/ovnkube-controller/ovnkube-controller/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.558873561Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c954x/ovnkube-controller/ovnkube-controller/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.558925172Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c954x/ovnkube-controller/ovnkube-controller/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.560983444Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c954x/ovnkube-controller/ovnkube-controller/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.561061616Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c954x/ovnkube-controller/ovnkube-controller/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.561110066Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c954x/sbdb/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.561144197Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c954x/sbdb/sbdb/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.561156518Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c954x/sbdb/sbdb/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.561190548Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c954x/sbdb/sbdb/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.561318961Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c954x/sbdb/sbdb/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.561405912Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c954x/sbdb/sbdb/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.561452544Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-kxrll/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.561543555Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-kxrll/ovnkube-node-kxrll.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.561718739Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-kxrll/kube-rbac-proxy-node/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.561732009Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-kxrll/kube-rbac-proxy-node/kube-rbac-proxy-node/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.561736879Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-kxrll/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.56176472Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-kxrll/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.561924393Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-kxrll/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.562041416Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-kxrll/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.562086257Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-kxrll/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.562099107Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-kxrll/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.562103467Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-kxrll/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.562174048Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-kxrll/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.562323871Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-kxrll/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.562423813Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-kxrll/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.562466904Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-kxrll/nbdb/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.562475064Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-kxrll/nbdb/nbdb/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.562479154Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-kxrll/nbdb/nbdb/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.562524645Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-kxrll/nbdb/nbdb/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.562652738Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-kxrll/nbdb/nbdb/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.56273469Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-kxrll/nbdb/nbdb/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.56276964Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-kxrll/northd/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.562779451Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-kxrll/northd/northd/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.562783691Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-kxrll/northd/northd/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.562834421Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-kxrll/northd/northd/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.562958384Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-kxrll/northd/northd/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.563041016Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-kxrll/northd/northd/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.563085377Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-kxrll/ovn-acl-logging/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.563095987Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-kxrll/ovn-acl-logging/ovn-acl-logging/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.563100007Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-kxrll/ovn-acl-logging/ovn-acl-logging/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.563137858Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-kxrll/ovn-acl-logging/ovn-acl-logging/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.56326193Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-kxrll/ovn-acl-logging/ovn-acl-logging/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.563334052Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-kxrll/ovn-controller/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.563345602Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-kxrll/ovn-controller/ovn-controller/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.563351502Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-kxrll/ovn-controller/ovn-controller/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.563394823Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-kxrll/ovn-controller/ovn-controller/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.563719199Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-kxrll/ovn-controller/ovn-controller/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.563798441Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-kxrll/ovn-controller/ovn-controller/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.563839402Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-kxrll/ovnkube-controller/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.563849262Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-kxrll/ovnkube-controller/ovnkube-controller/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.563854282Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-kxrll/ovnkube-controller/ovnkube-controller/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.563906383Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-kxrll/ovnkube-controller/ovnkube-controller/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.565782531Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-kxrll/ovnkube-controller/ovnkube-controller/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.565873453Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-kxrll/ovnkube-controller/ovnkube-controller/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.565919964Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-kxrll/sbdb/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.565930294Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-kxrll/sbdb/sbdb/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.565935295Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-kxrll/sbdb/sbdb/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.565986025Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-kxrll/sbdb/sbdb/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.566107548Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-kxrll/sbdb/sbdb/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.56623017Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-kxrll/sbdb/sbdb/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.566258661Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-qbd6q/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.566352173Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-qbd6q/ovnkube-node-qbd6q.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.566526107Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-qbd6q/kube-rbac-proxy-node/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.566538017Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-qbd6q/kube-rbac-proxy-node/kube-rbac-proxy-node/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.566544907Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-qbd6q/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.566617118Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-qbd6q/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.566770872Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-qbd6q/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.566837303Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-qbd6q/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.566886844Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-qbd6q/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.566895774Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-qbd6q/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.566900104Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-qbd6q/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.566950775Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-qbd6q/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.567097408Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-qbd6q/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.56721383Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-qbd6q/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.567254481Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-qbd6q/nbdb/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.567268562Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-qbd6q/nbdb/nbdb/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.567274992Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-qbd6q/nbdb/nbdb/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.567306602Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-qbd6q/nbdb/nbdb/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.567434785Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-qbd6q/nbdb/nbdb/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.567536017Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-qbd6q/nbdb/nbdb/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.567577658Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-qbd6q/northd/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.567585958Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-qbd6q/northd/northd/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.567591658Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-qbd6q/northd/northd/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.567634439Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-qbd6q/northd/northd/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.567764182Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-qbd6q/northd/northd/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.567832873Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-qbd6q/northd/northd/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.567887694Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-qbd6q/ovn-acl-logging/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.567896464Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-qbd6q/ovn-acl-logging/ovn-acl-logging/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.567902675Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-qbd6q/ovn-acl-logging/ovn-acl-logging/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.567942985Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-qbd6q/ovn-acl-logging/ovn-acl-logging/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.568058888Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-qbd6q/ovn-acl-logging/ovn-acl-logging/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.568115629Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-qbd6q/ovn-controller/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.568147019Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-qbd6q/ovn-controller/ovn-controller/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.56815878Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-qbd6q/ovn-controller/ovn-controller/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.568278452Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-qbd6q/ovn-controller/ovn-controller/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.568603279Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-qbd6q/ovn-controller/ovn-controller/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.568697911Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-qbd6q/ovn-controller/ovn-controller/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.568733141Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-qbd6q/ovnkube-controller/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.568742472Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-qbd6q/ovnkube-controller/ovnkube-controller/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.568747422Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-qbd6q/ovnkube-controller/ovnkube-controller/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.568798483Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-qbd6q/ovnkube-controller/ovnkube-controller/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.570710641Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-qbd6q/ovnkube-controller/ovnkube-controller/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.570796203Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-qbd6q/ovnkube-controller/ovnkube-controller/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.570907335Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-qbd6q/sbdb/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.570918126Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-qbd6q/sbdb/sbdb/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.570923836Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-qbd6q/sbdb/sbdb/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.571078179Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-qbd6q/sbdb/sbdb/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.571209602Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-qbd6q/sbdb/sbdb/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.571287803Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-qbd6q/sbdb/sbdb/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.571334754Z namespaces/openshift-ovn-kubernetes/policy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.571410116Z namespaces/openshift-ovn-kubernetes/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.571535558Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.571545668Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.571576849Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.571698461Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/openshift-ovn-kubernetes-nodes-identity-limited.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.571847815Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/prometheus-k8s.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.571901666Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.571947597Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.572056469Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/openshift-ovn-kubernetes-node-limited.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.572169241Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/prometheus-k8s.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.572234532Z namespaces/openshift-ovn-kubernetes/route.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.572321474Z namespaces/openshift-ovn-kubernetes/route.openshift.io/routes.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.572377455Z namespaces/openshift-route-controller-manager/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.572385715Z namespaces/openshift-route-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.572389706Z namespaces/openshift-route-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.572475227Z namespaces/openshift-route-controller-manager/coordination.k8s.io/leases/openshift-route-controllers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.572507518Z namespaces/openshift-route-controller-manager/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.572513318Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.572560799Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.572760893Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.572983027Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.573199612Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.573415896Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.573627171Z namespaces/openshift-service-ca-operator/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.573665221Z namespaces/openshift-service-ca-operator/openshift-service-ca-operator.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.573741623Z namespaces/openshift-service-ca-operator/apps.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.573759063Z namespaces/openshift-service-ca-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.573825375Z namespaces/openshift-service-ca-operator/apps/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.573863275Z namespaces/openshift-service-ca-operator/apps/daemonsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.573940697Z namespaces/openshift-service-ca-operator/apps/deployments.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.574062749Z namespaces/openshift-service-ca-operator/apps/replicasets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.574172562Z namespaces/openshift-service-ca-operator/apps/statefulsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.574222703Z namespaces/openshift-service-ca-operator/autoscaling/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.574273324Z namespaces/openshift-service-ca-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.574321115Z namespaces/openshift-service-ca-operator/batch/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.574342045Z namespaces/openshift-service-ca-operator/batch/cronjobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.574431507Z namespaces/openshift-service-ca-operator/batch/jobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.574473838Z namespaces/openshift-service-ca-operator/build.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.574516889Z namespaces/openshift-service-ca-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.574604971Z namespaces/openshift-service-ca-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.574650541Z namespaces/openshift-service-ca-operator/coordination.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.574662082Z namespaces/openshift-service-ca-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.574692752Z namespaces/openshift-service-ca-operator/coordination.k8s.io/leases/service-ca-operator-lock.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.574747813Z namespaces/openshift-service-ca-operator/core/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.574779334Z namespaces/openshift-service-ca-operator/core/configmaps.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.574894546Z namespaces/openshift-service-ca-operator/core/endpoints.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.574977748Z namespaces/openshift-service-ca-operator/core/events.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.575198703Z namespaces/openshift-service-ca-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.575277474Z namespaces/openshift-service-ca-operator/core/pods.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.575416007Z namespaces/openshift-service-ca-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.575539179Z namespaces/openshift-service-ca-operator/core/secrets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.575715913Z namespaces/openshift-service-ca-operator/core/services.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.575777424Z namespaces/openshift-service-ca-operator/discovery.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.575821915Z namespaces/openshift-service-ca-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.575867326Z namespaces/openshift-service-ca-operator/image.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.575914017Z namespaces/openshift-service-ca-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.575961088Z namespaces/openshift-service-ca-operator/k8s.ovn.org/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.575993029Z namespaces/openshift-service-ca-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.57607334Z namespaces/openshift-service-ca-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.576192533Z namespaces/openshift-service-ca-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.576237364Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.576280105Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.576331786Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.576373776Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/servicemonitors/service-ca-operator.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.576423657Z namespaces/openshift-service-ca-operator/networking.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.576458218Z namespaces/openshift-service-ca-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.576505739Z namespaces/openshift-service-ca-operator/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.576513299Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.5765547Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.576706953Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.576874646Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.5770238Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.577249444Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.577424288Z namespaces/openshift-service-ca-operator/pods/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.577436878Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d6c75f84f-d65fd/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.577450078Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d6c75f84f-d65fd/service-ca-operator-5d6c75f84f-d65fd.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.57753264Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d6c75f84f-d65fd/service-ca-operator/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.57754097Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d6c75f84f-d65fd/service-ca-operator/service-ca-operator/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.57754536Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d6c75f84f-d65fd/service-ca-operator/service-ca-operator/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.577571421Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d6c75f84f-d65fd/service-ca-operator/service-ca-operator/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.577771825Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d6c75f84f-d65fd/service-ca-operator/service-ca-operator/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.577963009Z namespaces/openshift-service-ca-operator/policy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.577988089Z namespaces/openshift-service-ca-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.57804359Z namespaces/openshift-service-ca-operator/route.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.578079701Z namespaces/openshift-service-ca-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.578143352Z namespaces/openshift-service-ca/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.578197223Z namespaces/openshift-service-ca/openshift-service-ca.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.578261615Z namespaces/openshift-service-ca/apps.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.578293145Z namespaces/openshift-service-ca/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.578362267Z namespaces/openshift-service-ca/apps/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.578382987Z namespaces/openshift-service-ca/apps/daemonsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.578469269Z namespaces/openshift-service-ca/apps/deployments.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.57855276Z namespaces/openshift-service-ca/apps/replicasets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.578658123Z namespaces/openshift-service-ca/apps/statefulsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.578698943Z namespaces/openshift-service-ca/autoscaling/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.578736274Z namespaces/openshift-service-ca/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.578790105Z namespaces/openshift-service-ca/batch/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.578828046Z namespaces/openshift-service-ca/batch/cronjobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.578899547Z namespaces/openshift-service-ca/batch/jobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.578945168Z namespaces/openshift-service-ca/build.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.578984849Z namespaces/openshift-service-ca/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.579061851Z namespaces/openshift-service-ca/build.openshift.io/builds.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.579116192Z namespaces/openshift-service-ca/coordination.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.579155163Z namespaces/openshift-service-ca/coordination.k8s.io/leases/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.579196424Z namespaces/openshift-service-ca/coordination.k8s.io/leases/service-ca-controller-lock.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.579256005Z namespaces/openshift-service-ca/core/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.579293035Z namespaces/openshift-service-ca/core/configmaps.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.579399768Z namespaces/openshift-service-ca/core/endpoints.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.579471709Z namespaces/openshift-service-ca/core/events.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.579579151Z namespaces/openshift-service-ca/core/persistentvolumeclaims.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.579657083Z namespaces/openshift-service-ca/core/pods.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.579787616Z namespaces/openshift-service-ca/core/replicationcontrollers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.579893628Z namespaces/openshift-service-ca/core/secrets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.580062001Z namespaces/openshift-service-ca/core/services.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.580104402Z namespaces/openshift-service-ca/discovery.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.580169653Z namespaces/openshift-service-ca/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.580232235Z namespaces/openshift-service-ca/image.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.580279666Z namespaces/openshift-service-ca/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.580318516Z namespaces/openshift-service-ca/k8s.ovn.org/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.580362397Z namespaces/openshift-service-ca/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.580450379Z namespaces/openshift-service-ca/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.58052168Z namespaces/openshift-service-ca/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.580562571Z namespaces/openshift-service-ca/monitoring.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.580604852Z namespaces/openshift-service-ca/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.580654373Z namespaces/openshift-service-ca/networking.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.580684314Z namespaces/openshift-service-ca/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.580735895Z namespaces/openshift-service-ca/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.580750375Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.580778316Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.580927929Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.581103612Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.581312396Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.581516621Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.581694074Z namespaces/openshift-service-ca/pods/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.581708115Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-fff8d/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.581732645Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-fff8d/service-ca-658d7b7c78-fff8d.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.581810517Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-fff8d/service-ca-controller/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.581819297Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-fff8d/service-ca-controller/service-ca-controller/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.581823577Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-fff8d/service-ca-controller/service-ca-controller/logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.581851707Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-fff8d/service-ca-controller/service-ca-controller/logs/current.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.582092623Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-fff8d/service-ca-controller/service-ca-controller/logs/previous.insecure.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.582177804Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-fff8d/service-ca-controller/service-ca-controller/logs/previous.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.582215285Z namespaces/openshift-service-ca/policy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.582264126Z namespaces/openshift-service-ca/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.582313727Z namespaces/openshift-service-ca/route.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.582332217Z namespaces/openshift-service-ca/route.openshift.io/routes.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.582383298Z namespaces/openshift-user-workload-monitoring/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.582428549Z namespaces/openshift-user-workload-monitoring/openshift-user-workload-monitoring.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.582492311Z namespaces/openshift-user-workload-monitoring/apps.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.582529511Z namespaces/openshift-user-workload-monitoring/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.582581302Z namespaces/openshift-user-workload-monitoring/apps/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.582630213Z namespaces/openshift-user-workload-monitoring/apps/daemonsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.582721615Z namespaces/openshift-user-workload-monitoring/apps/deployments.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.582797667Z namespaces/openshift-user-workload-monitoring/apps/replicasets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.582873838Z namespaces/openshift-user-workload-monitoring/apps/statefulsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.582905649Z namespaces/openshift-user-workload-monitoring/autoscaling/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.5829521Z namespaces/openshift-user-workload-monitoring/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.582989571Z namespaces/openshift-user-workload-monitoring/batch/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.583038282Z namespaces/openshift-user-workload-monitoring/batch/cronjobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.583118093Z namespaces/openshift-user-workload-monitoring/batch/jobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.583192145Z namespaces/openshift-user-workload-monitoring/build.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.583237816Z namespaces/openshift-user-workload-monitoring/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.583312257Z namespaces/openshift-user-workload-monitoring/build.openshift.io/builds.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.583345618Z namespaces/openshift-user-workload-monitoring/core/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.583391639Z namespaces/openshift-user-workload-monitoring/core/configmaps.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.583491741Z namespaces/openshift-user-workload-monitoring/core/endpoints.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.583572593Z namespaces/openshift-user-workload-monitoring/core/events.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.583654454Z namespaces/openshift-user-workload-monitoring/core/persistentvolumeclaims.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.583730926Z namespaces/openshift-user-workload-monitoring/core/pods.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.583810157Z namespaces/openshift-user-workload-monitoring/core/replicationcontrollers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.58393152Z namespaces/openshift-user-workload-monitoring/core/secrets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.584172354Z namespaces/openshift-user-workload-monitoring/core/services.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.584250256Z namespaces/openshift-user-workload-monitoring/discovery.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.584294557Z namespaces/openshift-user-workload-monitoring/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.584341438Z namespaces/openshift-user-workload-monitoring/image.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.584390989Z namespaces/openshift-user-workload-monitoring/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.58443457Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.584480091Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.584539072Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.584636944Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.584677145Z namespaces/openshift-user-workload-monitoring/monitoring.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.584722636Z namespaces/openshift-user-workload-monitoring/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.584772017Z namespaces/openshift-user-workload-monitoring/networking.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.584810367Z namespaces/openshift-user-workload-monitoring/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.584862339Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.584870009Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.584898219Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.585038052Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.585260587Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.585392079Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.585586193Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.585891509Z namespaces/openshift-user-workload-monitoring/policy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.58592221Z namespaces/openshift-user-workload-monitoring/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.585972701Z namespaces/openshift-user-workload-monitoring/route.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.586021432Z namespaces/openshift-user-workload-monitoring/route.openshift.io/routes.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.586069953Z namespaces/openshift/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.586112384Z namespaces/openshift/openshift.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.586188746Z namespaces/openshift/apps.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.586236136Z namespaces/openshift/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.586276597Z namespaces/openshift/apps/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.586327008Z namespaces/openshift/apps/daemonsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.58640194Z namespaces/openshift/apps/deployments.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.586476231Z namespaces/openshift/apps/replicasets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.586550503Z namespaces/openshift/apps/statefulsets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.586617894Z namespaces/openshift/autoscaling/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.586662035Z namespaces/openshift/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.586716056Z namespaces/openshift/batch/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.586754067Z namespaces/openshift/batch/cronjobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.586870459Z namespaces/openshift/batch/jobs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.58691541Z namespaces/openshift/build.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.586948741Z namespaces/openshift/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.587034853Z namespaces/openshift/build.openshift.io/builds.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.587087464Z namespaces/openshift/core/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.587149465Z namespaces/openshift/core/configmaps.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.587268157Z namespaces/openshift/core/endpoints.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.587341679Z namespaces/openshift/core/events.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.58742396Z namespaces/openshift/core/persistentvolumeclaims.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.587503372Z namespaces/openshift/core/pods.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.587584354Z namespaces/openshift/core/replicationcontrollers.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.587706786Z namespaces/openshift/core/secrets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.587863269Z namespaces/openshift/core/services.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.587909061Z namespaces/openshift/discovery.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.587953291Z namespaces/openshift/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.587992942Z namespaces/openshift/image.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.588040993Z namespaces/openshift/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.589251098Z namespaces/openshift/image.openshift.io/imagestreams/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.589288209Z namespaces/openshift/image.openshift.io/imagestreams/cli-artifacts.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.58938035Z namespaces/openshift/image.openshift.io/imagestreams/cli.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.589463722Z namespaces/openshift/image.openshift.io/imagestreams/dotnet-runtime.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.589566404Z namespaces/openshift/image.openshift.io/imagestreams/dotnet.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.589659516Z namespaces/openshift/image.openshift.io/imagestreams/driver-toolkit.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.589751188Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-eap-openshift-java11.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.589912241Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-eap-openshift.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.590077174Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-java-openshift.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.590246038Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-java11-openshift.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.590314489Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-karaf-openshift-jdk11.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.590411381Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-karaf-openshift.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.590529734Z namespaces/openshift/image.openshift.io/imagestreams/golang.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.590638436Z namespaces/openshift/image.openshift.io/imagestreams/httpd.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.590786879Z namespaces/openshift/image.openshift.io/imagestreams/installer-artifacts.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.590885041Z namespaces/openshift/image.openshift.io/imagestreams/installer.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.590977533Z namespaces/openshift/image.openshift.io/imagestreams/java-runtime.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.591078135Z namespaces/openshift/image.openshift.io/imagestreams/java.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.591229448Z namespaces/openshift/image.openshift.io/imagestreams/jboss-datagrid73-openshift.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.59135033Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp3-openjdk11-openshift.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.591451292Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp3-openjdk11-runtime-openshift.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.591544674Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp4-openjdk11-openshift.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.591645116Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp4-openjdk11-runtime-openshift.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.591733858Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk11-openshift.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.5918287Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk11-runtime-openshift.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.591922402Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk8-openshift.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.592021124Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk8-runtime-openshift.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.592107876Z namespaces/openshift/image.openshift.io/imagestreams/jboss-webserver57-openjdk11-tomcat9-openshift-ubi8.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.59231686Z namespaces/openshift/image.openshift.io/imagestreams/jboss-webserver57-openjdk8-tomcat9-openshift-ubi8.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.592408912Z namespaces/openshift/image.openshift.io/imagestreams/jenkins-agent-base.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.592511804Z namespaces/openshift/image.openshift.io/imagestreams/jenkins.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.592610736Z namespaces/openshift/image.openshift.io/imagestreams/mariadb.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.592724548Z namespaces/openshift/image.openshift.io/imagestreams/must-gather.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.59281099Z namespaces/openshift/image.openshift.io/imagestreams/mysql.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.592917142Z namespaces/openshift/image.openshift.io/imagestreams/network-tools.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.593002054Z namespaces/openshift/image.openshift.io/imagestreams/nginx.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.593139477Z namespaces/openshift/image.openshift.io/imagestreams/nodejs.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.593283939Z namespaces/openshift/image.openshift.io/imagestreams/oauth-proxy.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.593383981Z namespaces/openshift/image.openshift.io/imagestreams/openjdk-11-rhel7.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.593522284Z namespaces/openshift/image.openshift.io/imagestreams/perl.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.593638507Z namespaces/openshift/image.openshift.io/imagestreams/php.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.593745409Z namespaces/openshift/image.openshift.io/imagestreams/postgresql.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.593874151Z namespaces/openshift/image.openshift.io/imagestreams/postgresql13-for-sso75-openshift-rhel8.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.593962193Z namespaces/openshift/image.openshift.io/imagestreams/postgresql13-for-sso76-openshift-rhel8.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.594058415Z namespaces/openshift/image.openshift.io/imagestreams/python.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.594204658Z namespaces/openshift/image.openshift.io/imagestreams/redhat-openjdk18-openshift.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.594330971Z namespaces/openshift/image.openshift.io/imagestreams/redis.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.594443273Z namespaces/openshift/image.openshift.io/imagestreams/ruby.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.594548545Z namespaces/openshift/image.openshift.io/imagestreams/sso75-openshift-rhel8.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.594649507Z namespaces/openshift/image.openshift.io/imagestreams/sso76-openshift-rhel8.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.594832001Z namespaces/openshift/image.openshift.io/imagestreams/tests.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.594913093Z namespaces/openshift/image.openshift.io/imagestreams/tools.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.595034625Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-11-runtime.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.595175648Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-11.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.595314501Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-17-runtime.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.595420683Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-17.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.595539485Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-21-runtime.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.595624227Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-21.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.595734669Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-8-runtime.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.595938584Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-8.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.596017825Z namespaces/openshift/k8s.ovn.org/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.596066576Z namespaces/openshift/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.596153138Z namespaces/openshift/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.59625427Z namespaces/openshift/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.59627302Z namespaces/openshift/monitoring.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.596329321Z namespaces/openshift/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.596372252Z namespaces/openshift/networking.k8s.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.596416983Z namespaces/openshift/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.596454704Z namespaces/openshift/operators.coreos.com/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.596466204Z namespaces/openshift/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.596497285Z namespaces/openshift/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.596641508Z namespaces/openshift/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.596806721Z namespaces/openshift/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.596938874Z namespaces/openshift/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.597159398Z namespaces/openshift/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.597363112Z namespaces/openshift/policy/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.597379093Z namespaces/openshift/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.597426743Z namespaces/openshift/route.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.597470904Z namespaces/openshift/route.openshift.io/routes.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.597507595Z namespaces/openshift/template.openshift.io/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.597515385Z namespaces/openshift/template.openshift.io/templates/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.597559426Z namespaces/openshift/template.openshift.io/templates/cache-service.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.597661878Z namespaces/openshift/template.openshift.io/templates/cakephp-mysql-example.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.597780541Z namespaces/openshift/template.openshift.io/templates/cakephp-mysql-persistent.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.597882873Z namespaces/openshift/template.openshift.io/templates/dancer-mysql-example.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.597989755Z namespaces/openshift/template.openshift.io/templates/dancer-mysql-persistent.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.598178789Z namespaces/openshift/template.openshift.io/templates/datagrid-service.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.598287091Z namespaces/openshift/template.openshift.io/templates/django-psql-example.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.598398793Z namespaces/openshift/template.openshift.io/templates/django-psql-persistent.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.598513686Z namespaces/openshift/template.openshift.io/templates/eap-xp3-basic-s2i.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.598651688Z namespaces/openshift/template.openshift.io/templates/eap-xp4-basic-s2i.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.598768221Z namespaces/openshift/template.openshift.io/templates/eap74-basic-s2i.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.598885023Z namespaces/openshift/template.openshift.io/templates/eap74-https-s2i.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.598995526Z namespaces/openshift/template.openshift.io/templates/eap74-sso-s2i.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.599143818Z namespaces/openshift/template.openshift.io/templates/httpd-example.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.599268651Z namespaces/openshift/template.openshift.io/templates/jenkins-ephemeral-monitored.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.599388873Z namespaces/openshift/template.openshift.io/templates/jenkins-ephemeral.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.599496376Z namespaces/openshift/template.openshift.io/templates/jenkins-persistent-monitored.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.599613748Z namespaces/openshift/template.openshift.io/templates/jenkins-persistent.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.59971926Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk11-tomcat9-ubi8-basic-s2i.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.599813792Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk11-tomcat9-ubi8-https-s2i.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.599924434Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk8-tomcat9-ubi8-basic-s2i.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.600027416Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk8-tomcat9-ubi8-https-s2i.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.600151509Z namespaces/openshift/template.openshift.io/templates/mariadb-ephemeral.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.600271681Z namespaces/openshift/template.openshift.io/templates/mariadb-persistent.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.600372043Z namespaces/openshift/template.openshift.io/templates/mysql-ephemeral.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.600470365Z namespaces/openshift/template.openshift.io/templates/mysql-persistent.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.600562877Z namespaces/openshift/template.openshift.io/templates/nginx-example.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.600668729Z namespaces/openshift/template.openshift.io/templates/nodejs-postgresql-example.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.600769882Z namespaces/openshift/template.openshift.io/templates/nodejs-postgresql-persistent.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.600924754Z namespaces/openshift/template.openshift.io/templates/openjdk-web-basic-s2i.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.601040407Z namespaces/openshift/template.openshift.io/templates/postgresql-ephemeral.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.601144249Z namespaces/openshift/template.openshift.io/templates/postgresql-persistent.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.601378324Z namespaces/openshift/template.openshift.io/templates/rails-pgsql-persistent.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.601495836Z namespaces/openshift/template.openshift.io/templates/rails-postgresql-example.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.601625189Z namespaces/openshift/template.openshift.io/templates/react-web-app-example.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.60170441Z namespaces/openshift/template.openshift.io/templates/redis-ephemeral.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.601810452Z namespaces/openshift/template.openshift.io/templates/redis-persistent.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.601921775Z namespaces/openshift/template.openshift.io/templates/s2i-fuse712-spring-boot-2-camel-rest-3scale.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.602038757Z namespaces/openshift/template.openshift.io/templates/s2i-fuse712-spring-boot-2-camel-xml.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.602136429Z namespaces/openshift/template.openshift.io/templates/s2i-fuse712-spring-boot-2-camel.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.602277582Z namespaces/openshift/template.openshift.io/templates/sso75-https.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.602382944Z namespaces/openshift/template.openshift.io/templates/sso75-ocp4-x509-https.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.602501897Z namespaces/openshift/template.openshift.io/templates/sso75-ocp4-x509-postgresql-persistent.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.602634419Z namespaces/openshift/template.openshift.io/templates/sso75-postgresql-persistent.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.602747062Z namespaces/openshift/template.openshift.io/templates/sso75-postgresql.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.602877164Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-https.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.603023587Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-postgresql-persistent.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.60314351Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-postgresql.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.603301463Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-x509-https.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.603412265Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-x509-postgresql-persistent.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.603526698Z network_logs/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.603566168Z network_logs/cluster_scale [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.60364751Z network_logs/ippools.whereabouts.cni.cncf.io [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.603754712Z network_logs/multi-networkpolicy [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.603895575Z network_logs/net-attach-def [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.603979456Z network_logs/overlappingrangeipreservations.whereabouts.cni.cncf.io [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.60413957Z network_logs/ovn_kubernetes_top_pods [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.604243642Z network_logs/ovnk_database_store.tar.gz [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.606466407Z nodes/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.606488567Z nodes/debug [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.60660443Z nodes/ip-10-0-135-188.ec2.internal/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.6066313Z nodes/ip-10-0-135-188.ec2.internal/cpu_affinities.json [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.606952377Z nodes/ip-10-0-135-188.ec2.internal/dmesg [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.607245213Z nodes/ip-10-0-135-188.ec2.internal/ethtool_channels [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.607320874Z nodes/ip-10-0-135-188.ec2.internal/ethtool_features [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.607415856Z nodes/ip-10-0-135-188.ec2.internal/ip-10-0-135-188.ec2.internal_logs_kubelet.gz [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.607838435Z nodes/ip-10-0-135-188.ec2.internal/irq_affinities.json [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.607994838Z nodes/ip-10-0-135-188.ec2.internal/lscpu [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.608149081Z nodes/ip-10-0-135-188.ec2.internal/lspci [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.608249653Z nodes/ip-10-0-135-188.ec2.internal/podresources.json [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.608357056Z nodes/ip-10-0-135-188.ec2.internal/pods_info.json [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.608467598Z nodes/ip-10-0-135-188.ec2.internal/proc_cmdline [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.608547929Z nodes/ip-10-0-135-188.ec2.internal/sysinfo.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.610318485Z nodes/ip-10-0-135-188.ec2.internal/sysinfo.tgz [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.610482728Z nodes/ip-10-0-135-20.ec2.internal/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.610520609Z nodes/ip-10-0-135-20.ec2.internal/cpu_affinities.json [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.610781455Z nodes/ip-10-0-135-20.ec2.internal/dmesg [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.61101761Z nodes/ip-10-0-135-20.ec2.internal/ethtool_channels [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.611096831Z nodes/ip-10-0-135-20.ec2.internal/ethtool_features [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.611240274Z nodes/ip-10-0-135-20.ec2.internal/ip-10-0-135-20.ec2.internal_logs_kubelet.gz [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.611645792Z nodes/ip-10-0-135-20.ec2.internal/irq_affinities.json [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.611788525Z nodes/ip-10-0-135-20.ec2.internal/lscpu [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.611909948Z nodes/ip-10-0-135-20.ec2.internal/lspci [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.611994599Z nodes/ip-10-0-135-20.ec2.internal/podresources.json [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.612107592Z nodes/ip-10-0-135-20.ec2.internal/pods_info.json [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.612250034Z nodes/ip-10-0-135-20.ec2.internal/proc_cmdline [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.612325246Z nodes/ip-10-0-135-20.ec2.internal/sysinfo.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.61397807Z nodes/ip-10-0-135-20.ec2.internal/sysinfo.tgz [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.614146403Z nodes/ip-10-0-140-52.ec2.internal/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.614200984Z nodes/ip-10-0-140-52.ec2.internal/cpu_affinities.json [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.614525141Z nodes/ip-10-0-140-52.ec2.internal/dmesg [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.614774646Z nodes/ip-10-0-140-52.ec2.internal/ethtool_channels [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.614848357Z nodes/ip-10-0-140-52.ec2.internal/ethtool_features [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.614931339Z nodes/ip-10-0-140-52.ec2.internal/ip-10-0-140-52.ec2.internal_logs_kubelet.gz [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.615502821Z nodes/ip-10-0-140-52.ec2.internal/irq_affinities.json [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.615656673Z nodes/ip-10-0-140-52.ec2.internal/lscpu [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.615782896Z nodes/ip-10-0-140-52.ec2.internal/lspci [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.615866218Z nodes/ip-10-0-140-52.ec2.internal/podresources.json [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.61598458Z nodes/ip-10-0-140-52.ec2.internal/pods_info.json [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.616101913Z nodes/ip-10-0-140-52.ec2.internal/proc_cmdline [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.616229075Z nodes/ip-10-0-140-52.ec2.internal/sysinfo.log [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.617887559Z nodes/ip-10-0-140-52.ec2.internal/sysinfo.tgz [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.618037252Z pod_network_connectivity_check/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.618075633Z pod_network_connectivity_check/podnetworkconnectivitychecks.yaml [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.618199935Z static-pods/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.618215465Z static-pods/kube-apiserver/ [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.626391291Z [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.626408742Z sent 72,415 bytes received 8,406,124 bytes 5,652,359.33 bytes/sec [must-gather] [must-gather-wlhgr] OUT 2026-07-28T18:56:54.626417942Z total size is 103,344,090 speedup is 12.19 [must-gather] [must-gather ] OUT 2026-07-28T18:56:54.926144354Z namespace/openshift-must-gather-ct488 deleted [must-gather] [must-gather] [must-gather] Reprinting Cluster State: [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: 331d3e0a-05b1-4db5-9f55-68440dff074b [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.26" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [git-push-artifacts] WORK_DIR: /workspace/odh-ci-artifacts [git-push-artifacts] REPO_PATH: opendatahub-io/odh-build-metadata [git-push-artifacts] REPO_BRANCH: ci-artifacts [git-push-artifacts] SPARSE_FILE_PATH: test-artifacts/docs [git-push-artifacts] SOURCE_PATH: /workspace/artifacts-dir [git-push-artifacts] DEST_PATH: test-artifacts/kserve-group-test-fspzw [git-push-artifacts] ALWAYS_PASS: false [git-push-artifacts] configuring gh token [git-push-artifacts] taking github token from Konflux bot [git-push-artifacts] Initialized empty Git repository in /workspace/odh-ci-artifacts/.git/ [git-push-artifacts] Using partial fetch with sparse checkout for: test-artifacts/docs [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] * [new branch] ci-artifacts -> origin/ci-artifacts [git-push-artifacts] Already on 'ci-artifacts' [git-push-artifacts] branch 'ci-artifacts' set up to track 'origin/ci-artifacts'. [git-push-artifacts] TASK_NAME=kserve-group-test-fspzw-e2e-llm-inference-service [git-push-artifacts] PIPELINERUN_NAME=kserve-group-test-fspzw [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] Already up to date. [git-push-artifacts] -rw-r--r--. 1 root 1001540000 16722253 Jul 28 18:58 /workspace/odh-ci-artifacts/test-artifacts/kserve-group-test-fspzw/e2e-llm-inference-service.tar.gz [git-push-artifacts] [ci-artifacts 62ef477] Updating CI Artifacts in e2e-llm-inference-service [git-push-artifacts] 1 file changed, 0 insertions(+), 0 deletions(-) [git-push-artifacts] create mode 100644 test-artifacts/kserve-group-test-fspzw/e2e-llm-inference-service.tar.gz [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] Already up to date. [git-push-artifacts] To https://github.com/opendatahub-io/odh-build-metadata.git [git-push-artifacts] 7d63cb8..62ef477 ci-artifacts -> ci-artifacts [fail-if-needed] Failing pipeline because deploy-and-e2e step failed container step-fail-if-needed has failed : [{"key":"StartedAt","value":"2026-07-28T18:58:43.617Z","type":3}]