task e2e-llm-inference-service has failed: "step-fail-if-needed" exited with code 1: Error [get-kubeconfig] Found kubeconfig secret: cluster-tptt8-admin-kubeconfig [get-kubeconfig] Wrote kubeconfig to /credentials/cluster-tptt8-kubeconfig [get-kubeconfig] Found admin password secret: cluster-tptt8-admin-password [get-kubeconfig] Retrieved username [get-kubeconfig] Wrote password to /credentials/cluster-tptt8-password [get-kubeconfig] API Server URL: https://a594ea6a0475f44dc85157d7be9b2269-4d6af399b31c8a77.elb.us-east-1.amazonaws.com:6443 [get-kubeconfig] Console URL: https://console-openshift-console.apps.be080280-9636-44bb-bf42-272b50be1c89.prod.konfluxeaas.com [clone-repo] test/tls-rbac-subject-consistency [clone-repo] https://github.com/mwaykole/kserve [clone-repo] Cloning into '/workspace/source'... [clone-repo] Updating files: 46% (1503/3251) Updating files: 47% (1528/3251) Updating files: 48% (1561/3251) Updating files: 49% (1593/3251) Updating files: 50% (1626/3251) Updating files: 51% (1659/3251) Updating files: 52% (1691/3251) Updating files: 53% (1724/3251) Updating files: 54% (1756/3251) Updating files: 55% (1789/3251) Updating files: 56% (1821/3251) Updating files: 57% (1854/3251) Updating files: 58% (1886/3251) Updating files: 59% (1919/3251) Updating files: 60% (1951/3251) Updating files: 61% (1984/3251) Updating files: 62% (2016/3251) Updating files: 63% (2049/3251) Updating files: 64% (2081/3251) Updating files: 65% (2114/3251) Updating files: 66% (2146/3251) Updating files: 67% (2179/3251) Updating files: 68% (2211/3251) Updating files: 69% (2244/3251) Updating files: 70% (2276/3251) Updating files: 71% (2309/3251) Updating files: 72% (2341/3251) Updating files: 73% (2374/3251) Updating files: 74% (2406/3251) Updating files: 75% (2439/3251) Updating files: 76% (2471/3251) Updating files: 77% (2504/3251) Updating files: 78% (2536/3251) Updating files: 79% (2569/3251) Updating files: 80% (2601/3251) Updating files: 81% (2634/3251) Updating files: 82% (2666/3251) Updating files: 83% (2699/3251) Updating files: 84% (2731/3251) Updating files: 85% (2764/3251) Updating files: 86% (2796/3251) Updating files: 87% (2829/3251) Updating files: 88% (2861/3251) Updating files: 89% (2894/3251) Updating files: 90% (2926/3251) Updating files: 91% (2959/3251) Updating files: 92% (2991/3251) Updating files: 93% (3024/3251) Updating files: 94% (3056/3251) Updating files: 95% (3089/3251) Updating files: 96% (3121/3251) Updating files: 97% (3154/3251) Updating files: 98% (3186/3251) Updating files: 99% (3219/3251) Updating files: 100% (3251/3251) Updating files: 100% (3251/3251), done. [e2e-llm-inference-service] + bash [e2e-llm-inference-service] + STATUS_FILE=/test-status/deploy-and-e2e-status [e2e-llm-inference-service] + echo failed [e2e-llm-inference-service] + COMPONENT_NAME=kserve-agent-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-agent-ci '.[$component_name].image' [e2e-llm-inference-service] + export KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:7789d265344a8abbbdf46d39305389c0ec3159d12d2d92e2c5830f8aeb79d1f1 [e2e-llm-inference-service] + KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:7789d265344a8abbbdf46d39305389c0ec3159d12d2d92e2c5830f8aeb79d1f1 [e2e-llm-inference-service] + COMPONENT_NAME=kserve-controller-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-controller-ci '.[$component_name].image' [e2e-llm-inference-service] + export KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:a00b96ab2f9202facf950a577d55a24338ee980c7c01fced43402b87d97964be [e2e-llm-inference-service] + KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:a00b96ab2f9202facf950a577d55a24338ee980c7c01fced43402b87d97964be [e2e-llm-inference-service] + COMPONENT_NAME=kserve-router-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-router-ci '.[$component_name].image' [e2e-llm-inference-service] + export KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:bcb85ef6ed85b7da131182bb5f673741b67400a2637d62d1dcee3f3c438674f0 [e2e-llm-inference-service] + KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:bcb85ef6ed85b7da131182bb5f673741b67400a2637d62d1dcee3f3c438674f0 [e2e-llm-inference-service] + COMPONENT_NAME=kserve-storage-initializer-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-storage-initializer-ci '.[$component_name].image' [e2e-llm-inference-service] + export STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:527f9c4ef06f4e1d255c6c4b87333be01c67680fe0b3417b8fb7aeb5d5b61bbf [e2e-llm-inference-service] + STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:527f9c4ef06f4e1d255c6c4b87333be01c67680fe0b3417b8fb7aeb5d5b61bbf [e2e-llm-inference-service] + COMPONENT_NAME=odh-kserve-llmisvc-controller-ci [e2e-llm-inference-service] ++ jq -r --arg component_name odh-kserve-llmisvc-controller-ci '.[$component_name].image' [e2e-llm-inference-service] + export LLMISVC_CONTROLLER_IMAGE=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:b2c041a1b3fc0af41ba7db1aab2006d510e30f800ca4162b46f8fde3a4726d62 [e2e-llm-inference-service] + LLMISVC_CONTROLLER_IMAGE=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:b2c041a1b3fc0af41ba7db1aab2006d510e30f800ca4162b46f8fde3a4726d62 [e2e-llm-inference-service] + ./test/scripts/openshift-ci/run-e2e-tests.sh 'llminferenceservice and cluster_cpu and not autoscaling and not tracing' 2 llm-d [e2e-llm-inference-service] INFERENCE_POOL_GROUP=inference.networking.k8s.io (detected from OCP 4.21.26) [e2e-llm-inference-service] Installing on cluster [e2e-llm-inference-service] Using namespace: kserve for KServe components [e2e-llm-inference-service] SKLEARN_IMAGE=quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1813 [e2e-llm-inference-service] OPT_125M_MODEL_URI=s3://example-models/facebook/opt-125m [e2e-llm-inference-service] ERROR_404_ISVC_IMAGE=quay.io/opendatahub/error-404-isvc:odh-pr-1813 [e2e-llm-inference-service] SUCCESS_200_ISVC_IMAGE=quay.io/opendatahub/success-200-isvc:odh-pr-1813 [e2e-llm-inference-service] [INFO] Installing Kustomize v5.8.1 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed Kustomize v5.8.1 to /workspace/source/bin/kustomize [e2e-llm-inference-service] v5.8.1 [e2e-llm-inference-service] make: Entering directory '/workspace/source' [e2e-llm-inference-service] [INFO] Installing yq v4.52.1 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed yq v4.52.1 to /workspace/source/bin/yq [e2e-llm-inference-service] yq (https://github.com/mikefarah/yq/) version v4.52.1 [e2e-llm-inference-service] make: Leaving directory '/workspace/source' [e2e-llm-inference-service] Installing KServe Python SDK ... [e2e-llm-inference-service] [INFO] Installing uv 0.7.8 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed uv 0.7.8 to /workspace/source/bin/uv [e2e-llm-inference-service] warning: Failed to read project metadata (No `pyproject.toml` found in current directory or any parent directory). Running `uv self version` for compatibility. This fallback will be removed in the future; pass `--preview` to force an error. [e2e-llm-inference-service] uv 0.7.8 [e2e-llm-inference-service] Creating virtual environment... [e2e-llm-inference-service] warning: virtualenv's `--clear` has no effect (uv always clears the virtual environment) [e2e-llm-inference-service] Using CPython 3.9.25 interpreter at: /usr/bin/python3 [e2e-llm-inference-service] Creating virtual environment at: .venv [e2e-llm-inference-service] /workspace/source [e2e-llm-inference-service] Using CPython 3.11.13 interpreter at: /usr/bin/python3.11 [e2e-llm-inference-service] Creating virtual environment at: .venv [e2e-llm-inference-service] Resolved 274 packages in 1ms [e2e-llm-inference-service] Building kserve @ file:///workspace/source/python/kserve [e2e-llm-inference-service] Downloading pandas (12.5MiB) [e2e-llm-inference-service] Downloading pydantic-core (2.0MiB) [e2e-llm-inference-service] Downloading kubernetes (1.9MiB) [e2e-llm-inference-service] Downloading uvloop (3.8MiB) [e2e-llm-inference-service] Downloading aiohttp (1.7MiB) [e2e-llm-inference-service] Downloading cryptography (4.5MiB) [e2e-llm-inference-service] Downloading botocore (12.9MiB) [e2e-llm-inference-service] Downloading numpy (15.7MiB) [e2e-llm-inference-service] Downloading black (1.6MiB) [e2e-llm-inference-service] Downloading portforward (3.9MiB) [e2e-llm-inference-service] Downloading mypy (17.2MiB) [e2e-llm-inference-service] Downloading pyarrow (40.1MiB) [e2e-llm-inference-service] Downloading grpcio (6.4MiB) [e2e-llm-inference-service] Downloading grpcio-tools (2.5MiB) [e2e-llm-inference-service] Downloading setuptools (1.2MiB) [e2e-llm-inference-service] Building timeout-sampler==1.0.3 [e2e-llm-inference-service] Building python-simple-logger==2.0.19 [e2e-llm-inference-service] Downloading aiohttp [e2e-llm-inference-service] Downloading black [e2e-llm-inference-service] Downloading pydantic-core [e2e-llm-inference-service] Downloading grpcio-tools [e2e-llm-inference-service] Downloading setuptools [e2e-llm-inference-service] Built python-simple-logger==2.0.19 [e2e-llm-inference-service] Downloading portforward [e2e-llm-inference-service] Downloading uvloop [e2e-llm-inference-service] Downloading cryptography [e2e-llm-inference-service] Downloading grpcio [e2e-llm-inference-service] Downloading kubernetes [e2e-llm-inference-service] Built timeout-sampler==1.0.3 [e2e-llm-inference-service] Downloading numpy [e2e-llm-inference-service] Built kserve @ file:///workspace/source/python/kserve [e2e-llm-inference-service] Downloading pandas [e2e-llm-inference-service] Downloading botocore [e2e-llm-inference-service] Downloading pyarrow [e2e-llm-inference-service] Downloading mypy [e2e-llm-inference-service] Prepared 101 packages in 1.84s [e2e-llm-inference-service] warning: Failed to hardlink files; falling back to full copy. This may lead to degraded performance. [e2e-llm-inference-service] If the cache and target directories are on different filesystems, hardlinking may not be supported. [e2e-llm-inference-service] If this is intentional, set `export UV_LINK_MODE=copy` or use `--link-mode=copy` to suppress this warning. [e2e-llm-inference-service] Installed 101 packages in 342ms [e2e-llm-inference-service] + aiohappyeyeballs==2.6.1 [e2e-llm-inference-service] + aiohttp==3.14.1 [e2e-llm-inference-service] + aiosignal==1.4.0 [e2e-llm-inference-service] + annotated-doc==0.0.4 [e2e-llm-inference-service] + annotated-types==0.7.0 [e2e-llm-inference-service] + anyio==4.9.0 [e2e-llm-inference-service] + attrs==25.3.0 [e2e-llm-inference-service] + avro==1.12.0 [e2e-llm-inference-service] + black==24.3.0 [e2e-llm-inference-service] + boto3==1.37.35 [e2e-llm-inference-service] + botocore==1.37.35 [e2e-llm-inference-service] + cachetools==5.5.2 [e2e-llm-inference-service] + certifi==2025.1.31 [e2e-llm-inference-service] + cffi==2.0.0 [e2e-llm-inference-service] + charset-normalizer==3.4.1 [e2e-llm-inference-service] + click==8.4.2 [e2e-llm-inference-service] + cloudevents==1.11.0 [e2e-llm-inference-service] + colorama==0.4.6 [e2e-llm-inference-service] + colorlog==6.10.1 [e2e-llm-inference-service] + coverage==7.8.0 [e2e-llm-inference-service] + cryptography==49.0.0 [e2e-llm-inference-service] + deprecation==2.1.0 [e2e-llm-inference-service] + durationpy==0.9 [e2e-llm-inference-service] + execnet==2.1.1 [e2e-llm-inference-service] + fastapi==0.136.3 [e2e-llm-inference-service] + frozenlist==1.5.0 [e2e-llm-inference-service] + google-auth==2.39.0 [e2e-llm-inference-service] + grpc-interceptor==0.15.4 [e2e-llm-inference-service] + grpcio==1.78.1 [e2e-llm-inference-service] + grpcio-testing==1.78.1 [e2e-llm-inference-service] + grpcio-tools==1.78.1 [e2e-llm-inference-service] + h11==0.16.0 [e2e-llm-inference-service] + httpcore==1.0.9 [e2e-llm-inference-service] + httptools==0.6.4 [e2e-llm-inference-service] + httpx==0.27.2 [e2e-llm-inference-service] + httpx-retries==0.4.5 [e2e-llm-inference-service] + idna==3.10 [e2e-llm-inference-service] + iniconfig==2.1.0 [e2e-llm-inference-service] + jinja2==3.1.6 [e2e-llm-inference-service] + jmespath==1.0.1 [e2e-llm-inference-service] + kserve==0.20.0rc0 (from file:///workspace/source/python/kserve) [e2e-llm-inference-service] + kubernetes==32.0.1 [e2e-llm-inference-service] + markupsafe==3.0.2 [e2e-llm-inference-service] + multidict==6.4.3 [e2e-llm-inference-service] + mypy==0.991 [e2e-llm-inference-service] + mypy-extensions==1.0.0 [e2e-llm-inference-service] + numpy==2.2.4 [e2e-llm-inference-service] + oauthlib==3.2.2 [e2e-llm-inference-service] + orjson==3.11.9 [e2e-llm-inference-service] + packaging==24.2 [e2e-llm-inference-service] + pandas==2.2.3 [e2e-llm-inference-service] + pathspec==0.12.1 [e2e-llm-inference-service] + platformdirs==4.3.7 [e2e-llm-inference-service] + pluggy==1.5.0 [e2e-llm-inference-service] + portforward==0.7.1 [e2e-llm-inference-service] + prometheus-client==0.21.1 [e2e-llm-inference-service] + propcache==0.3.1 [e2e-llm-inference-service] + protobuf==6.33.5 [e2e-llm-inference-service] + psutil==5.9.8 [e2e-llm-inference-service] + pyarrow==19.0.1 [e2e-llm-inference-service] + pyasn1==0.6.3 [e2e-llm-inference-service] + pyasn1-modules==0.4.2 [e2e-llm-inference-service] + pycparser==2.22 [e2e-llm-inference-service] + pydantic==2.12.4 [e2e-llm-inference-service] + pydantic-core==2.41.5 [e2e-llm-inference-service] + pyjwt==2.13.0 [e2e-llm-inference-service] + pytest==7.4.4 [e2e-llm-inference-service] + pytest-asyncio==0.23.8 [e2e-llm-inference-service] + pytest-cov==5.0.0 [e2e-llm-inference-service] + pytest-httpx==0.30.0 [e2e-llm-inference-service] + pytest-json-report==1.5.0 [e2e-llm-inference-service] + pytest-metadata==3.1.1 [e2e-llm-inference-service] + pytest-xdist==3.6.1 [e2e-llm-inference-service] + python-dateutil==2.9.0.post0 [e2e-llm-inference-service] + python-dotenv==1.1.0 [e2e-llm-inference-service] + python-multipart==0.0.32 [e2e-llm-inference-service] + python-simple-logger==2.0.19 [e2e-llm-inference-service] + pytz==2025.2 [e2e-llm-inference-service] + pyyaml==6.0.2 [e2e-llm-inference-service] + requests==2.32.3 [e2e-llm-inference-service] + requests-oauthlib==2.0.0 [e2e-llm-inference-service] + rsa==4.9.1 [e2e-llm-inference-service] + s3transfer==0.11.4 [e2e-llm-inference-service] + setuptools==78.1.0 [e2e-llm-inference-service] + six==1.17.0 [e2e-llm-inference-service] + sniffio==1.3.1 [e2e-llm-inference-service] + starlette==1.2.1 [e2e-llm-inference-service] + tabulate==0.9.0 [e2e-llm-inference-service] + timeout-sampler==1.0.3 [e2e-llm-inference-service] + timing-asgi==0.3.1 [e2e-llm-inference-service] + tomlkit==0.13.2 [e2e-llm-inference-service] + typing-extensions==4.15.0 [e2e-llm-inference-service] + typing-inspection==0.4.2 [e2e-llm-inference-service] + tzdata==2025.2 [e2e-llm-inference-service] + urllib3==2.7.0 [e2e-llm-inference-service] + uvicorn==0.34.1 [e2e-llm-inference-service] + uvloop==0.21.0 [e2e-llm-inference-service] + watchfiles==1.0.5 [e2e-llm-inference-service] + websocket-client==1.8.0 [e2e-llm-inference-service] + websockets==15.0.1 [e2e-llm-inference-service] + yarl==1.20.0 [e2e-llm-inference-service] Audited 1 package in 47ms [e2e-llm-inference-service] /workspace/source [e2e-llm-inference-service] [INFO] Installing Kustomize v5.8.1 for linux/amd64... [e2e-llm-inference-service] [INFO] Kustomize v5.8.1 is already installed in /workspace/source/bin (>= v5.8.1) [e2e-llm-inference-service] make: Entering directory '/workspace/source' [e2e-llm-inference-service] make: Leaving directory '/workspace/source' [e2e-llm-inference-service] Now using project "kserve" on server "https://a594ea6a0475f44dc85157d7be9b2269-4d6af399b31c8a77.elb.us-east-1.amazonaws.com:6443". [e2e-llm-inference-service] [e2e-llm-inference-service] You can add applications to this project with the 'new-app' command. For example, try: [e2e-llm-inference-service] [e2e-llm-inference-service] oc new-app rails-postgresql-example [e2e-llm-inference-service] [e2e-llm-inference-service] to build a new example application in Ruby. Or use kubectl to deploy a simple Kubernetes application: [e2e-llm-inference-service] [e2e-llm-inference-service] kubectl create deployment hello-node --image=registry.k8s.io/e2e-test-images/agnhost:2.43 -- /agnhost serve-hostname [e2e-llm-inference-service] [e2e-llm-inference-service] [INFO] Installing Kustomize v5.8.1 for linux/amd64... [e2e-llm-inference-service] [INFO] Kustomize v5.8.1 is already installed in /workspace/source/bin (>= v5.8.1) [e2e-llm-inference-service] make: Entering directory '/workspace/source' [e2e-llm-inference-service] make: Leaving directory '/workspace/source' [e2e-llm-inference-service] Creating namespace openshift-keda... [e2e-llm-inference-service] namespace/openshift-keda created [e2e-llm-inference-service] Namespace openshift-keda created/ensured. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Creating OperatorGroup openshift-keda... [e2e-llm-inference-service] operatorgroup.operators.coreos.com/openshift-keda created [e2e-llm-inference-service] OperatorGroup openshift-keda created/ensured. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Creating Subscription for openshift-custom-metrics-autoscaler-operator... [e2e-llm-inference-service] subscription.operators.coreos.com/openshift-custom-metrics-autoscaler-operator created [e2e-llm-inference-service] Subscription openshift-custom-metrics-autoscaler-operator created/ensured. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Waiting for openshift-custom-metrics-autoscaler-operator CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (0/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (5/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (10/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (15/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (20/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-2 found, but not yet Succeeded (Phase: Installing). Waiting... (25/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-2 found, but not yet Succeeded (Phase: Installing). Waiting... (30/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-2 found, but not yet Succeeded (Phase: Installing). Waiting... (35/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-2 found, but not yet Succeeded (Phase: Installing). Waiting... (40/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-2 found, but not yet Succeeded (Phase: Installing). Waiting... (45/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-2 is ready (Phase: Succeeded). [e2e-llm-inference-service] --- [e2e-llm-inference-service] Applying KedaController custom resource... [e2e-llm-inference-service] Warning: resource kedacontrollers/keda is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by oc apply. oc apply should only be used on resources created declaratively by either oc create --save-config or oc apply. The missing annotation will be patched automatically. [e2e-llm-inference-service] kedacontroller.keda.sh/keda configured [e2e-llm-inference-service] KedaController custom resource applied. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Allowing time for KEDA components to be provisioned by the operator ... [e2e-llm-inference-service] Waiting for KEDA Operator pod (selector: "app=keda-operator") to be ready in namespace openshift-keda... [e2e-llm-inference-service] Waiting for pod -l "app=keda-operator" in namespace "openshift-keda" to be created... [e2e-llm-inference-service] Pod -l "app=keda-operator" in namespace "openshift-keda" found. [e2e-llm-inference-service] Current pods for -l "app=keda-operator" in namespace "openshift-keda": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] keda-operator-77f648d694-c9zcq 1/1 Running 0 41s [e2e-llm-inference-service] Waiting up to 120s for pod(s) -l "app=keda-operator" in namespace "openshift-keda" to become ready... [e2e-llm-inference-service] pod/keda-operator-77f648d694-c9zcq condition met [e2e-llm-inference-service] Pod(s) -l "app=keda-operator" in namespace "openshift-keda" are ready. [e2e-llm-inference-service] KEDA Operator pod is ready. [e2e-llm-inference-service] Waiting for KEDA Metrics API Server pod (selector: "app=keda-metrics-apiserver") to be ready in namespace openshift-keda... [e2e-llm-inference-service] Waiting for pod -l "app=keda-metrics-apiserver" in namespace "openshift-keda" to be created... [e2e-llm-inference-service] Pod -l "app=keda-metrics-apiserver" in namespace "openshift-keda" found. [e2e-llm-inference-service] Current pods for -l "app=keda-metrics-apiserver" in namespace "openshift-keda": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] keda-metrics-apiserver-55557df8c-w7shf 1/1 Running 0 46s [e2e-llm-inference-service] Waiting up to 120s for pod(s) -l "app=keda-metrics-apiserver" in namespace "openshift-keda" to become ready... [e2e-llm-inference-service] pod/keda-metrics-apiserver-55557df8c-w7shf condition met [e2e-llm-inference-service] Pod(s) -l "app=keda-metrics-apiserver" in namespace "openshift-keda" are ready. [e2e-llm-inference-service] KEDA Metrics API Server pod is ready. [e2e-llm-inference-service] Waiting for KEDA Webhook pod (selector: "app=keda-admission-webhooks") to be ready in namespace openshift-keda... [e2e-llm-inference-service] Waiting for pod -l "app=keda-admission-webhooks" in namespace "openshift-keda" to be created... [e2e-llm-inference-service] Pod -l "app=keda-admission-webhooks" in namespace "openshift-keda" found. [e2e-llm-inference-service] Current pods for -l "app=keda-admission-webhooks" in namespace "openshift-keda": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] keda-admission-56f9798bb4-9sg4n 1/1 Running 0 50s [e2e-llm-inference-service] Waiting up to 120s for pod(s) -l "app=keda-admission-webhooks" in namespace "openshift-keda" to become ready... [e2e-llm-inference-service] pod/keda-admission-56f9798bb4-9sg4n condition met [e2e-llm-inference-service] Pod(s) -l "app=keda-admission-webhooks" in namespace "openshift-keda" are ready. [e2e-llm-inference-service] KEDA Webhook pod is ready. [e2e-llm-inference-service] --- [e2e-llm-inference-service] ✅ KEDA deployment script finished successfully. [e2e-llm-inference-service] KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:a00b96ab2f9202facf950a577d55a24338ee980c7c01fced43402b87d97964be [e2e-llm-inference-service] LLMISVC_CONTROLLER_IMAGE=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:b2c041a1b3fc0af41ba7db1aab2006d510e30f800ca4162b46f8fde3a4726d62 [e2e-llm-inference-service] KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:7789d265344a8abbbdf46d39305389c0ec3159d12d2d92e2c5830f8aeb79d1f1 [e2e-llm-inference-service] KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:bcb85ef6ed85b7da131182bb5f673741b67400a2637d62d1dcee3f3c438674f0 [e2e-llm-inference-service] STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:527f9c4ef06f4e1d255c6c4b87333be01c67680fe0b3417b8fb7aeb5d5b61bbf [e2e-llm-inference-service] Installing KServe via kustomize... [e2e-llm-inference-service] === Final params.env [e2e-llm-inference-service] kserve-controller=quay.io/opendatahub/kserve-controller@sha256:a00b96ab2f9202facf950a577d55a24338ee980c7c01fced43402b87d97964be [e2e-llm-inference-service] llmisvc-controller=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:b2c041a1b3fc0af41ba7db1aab2006d510e30f800ca4162b46f8fde3a4726d62 [e2e-llm-inference-service] kserve-agent=quay.io/opendatahub/kserve-agent@sha256:7789d265344a8abbbdf46d39305389c0ec3159d12d2d92e2c5830f8aeb79d1f1 [e2e-llm-inference-service] kserve-router=quay.io/opendatahub/kserve-router@sha256:bcb85ef6ed85b7da131182bb5f673741b67400a2637d62d1dcee3f3c438674f0 [e2e-llm-inference-service] kserve-storage-initializer=quay.io/opendatahub/kserve-storage-initializer@sha256:527f9c4ef06f4e1d255c6c4b87333be01c67680fe0b3417b8fb7aeb5d5b61bbf [e2e-llm-inference-service] kserve-llm-d=registry.redhat.io/rhaiis/vllm-cuda-rhel9@sha256:fc68d623d1bfc36c8cb2fe4a71f19c8578cfb420ce8ce07b20a02c1ee0be0cf3 [e2e-llm-inference-service] kserve-llm-d-inference-scheduler=quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] kserve-llm-d-routing-sidecar=quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.9.0 [e2e-llm-inference-service] kserve-llm-d-uds-tokenizer=quay.io/opendatahub/llm-d-kv-cache:v0.8.0 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda=registry.redhat.io/rhaiis/vllm-cuda-rhel9@sha256:fc68d623d1bfc36c8cb2fe4a71f19c8578cfb420ce8ce07b20a02c1ee0be0cf3 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda-fast-1=registry.redhat.io/rhaiis/vllm-cuda-rhel9@sha256:fc68d623d1bfc36c8cb2fe4a71f19c8578cfb420ce8ce07b20a02c1ee0be0cf3 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda-fast-2=registry.redhat.io/rhaiis/vllm-cuda-rhel9@sha256:fc68d623d1bfc36c8cb2fe4a71f19c8578cfb420ce8ce07b20a02c1ee0be0cf3 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda-fast-1-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda-fast-2-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-amd-rocm=registry.redhat.io/rhaiis/vllm-rocm-rhel9@sha256:d9a48add238cc095fa43eeee17c8c4d104de60c4dc623e0bc7f8c4b53b2b2e97 [e2e-llm-inference-service] kserve-llm-d-amd-rocm-fast-1=registry.redhat.io/rhaiis/vllm-rocm-rhel9@sha256:d9a48add238cc095fa43eeee17c8c4d104de60c4dc623e0bc7f8c4b53b2b2e97 [e2e-llm-inference-service] kserve-llm-d-amd-rocm-fast-2=registry.redhat.io/rhaiis/vllm-rocm-rhel9@sha256:d9a48add238cc095fa43eeee17c8c4d104de60c4dc623e0bc7f8c4b53b2b2e97 [e2e-llm-inference-service] kserve-llm-d-amd-rocm-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-amd-rocm-fast-1-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-amd-rocm-fast-2-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi=registry.redhat.io/rhaii-early-access/vllm-gaudi-rhel9:3.4.0-ea.2 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi-fast-1=registry.redhat.io/rhaii-early-access/vllm-gaudi-rhel9:3.4.0-ea.2 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi-fast-2=registry.redhat.io/rhaii-early-access/vllm-gaudi-rhel9:3.4.0-ea.2 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi-upstream-version=0.16.0 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi-fast-1-upstream-version=0.16.0 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi-fast-2-upstream-version=0.16.0 [e2e-llm-inference-service] kserve-llm-d-ibm-spyre=registry.redhat.io/rhaiis/vllm-spyre-rhel9@sha256:80ae3e435a5be2c1f117f36599103ab05357917dd6e37f0df6613cb3ac2c13ea [e2e-llm-inference-service] kserve-llm-d-ibm-spyre-fast-1=registry.redhat.io/rhaiis/vllm-spyre-rhel9@sha256:80ae3e435a5be2c1f117f36599103ab05357917dd6e37f0df6613cb3ac2c13ea [e2e-llm-inference-service] kserve-llm-d-ibm-spyre-fast-2=registry.redhat.io/rhaiis/vllm-spyre-rhel9@sha256:80ae3e435a5be2c1f117f36599103ab05357917dd6e37f0df6613cb3ac2c13ea [e2e-llm-inference-service] kserve-llm-d-ibm-spyre-upstream-version=0.10.2 [e2e-llm-inference-service] kserve-llm-d-ibm-spyre-fast-1-upstream-version=0.10.2 [e2e-llm-inference-service] kserve-llm-d-ibm-spyre-fast-2-upstream-version=0.10.2 [e2e-llm-inference-service] kserve-llm-d-latency-predictor-prediction=quay.io/opendatahub/odh-latency-predictor-prediction:odh-stable [e2e-llm-inference-service] kserve-llm-d-latency-predictor-training=quay.io/opendatahub/odh-latency-predictor-training:odh-stable [e2e-llm-inference-service] # TODO update when our changes are introduced in the official image [e2e-llm-inference-service] kube-rbac-proxy=quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3 [e2e-llm-inference-service] kserve-localmodel-controller=quay.io/opendatahub/odh-kserve-localmodel-controller:odh-master [e2e-llm-inference-service] kserve-localmodelnode-agent=quay.io/opendatahub/odh-kserve-localmodelnode-agent:odh-master [e2e-llm-inference-service] ovms-versioning-ubi-micro=registry.redhat.io/ubi9/ubi-micro@sha256:38e934147827349f2b8b11ac9c38d7be23cb8e29f128b1c46e5c7ae54a2d23cd [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/dscinitializations.dscinitialization.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencemodelrewrites.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceobjectives.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-llm-inference-service] Waiting for CRDs to be established... [e2e-llm-inference-service] Waiting for CRD inferenceservices.serving.kserve.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD inferenceservices.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io condition met [e2e-llm-inference-service] Waiting for CRD llminferenceserviceconfigs.serving.kserve.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD llminferenceserviceconfigs.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io condition met [e2e-llm-inference-service] Waiting for CRD clusterstoragecontainers.serving.kserve.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD clusterstoragecontainers.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io condition met [e2e-llm-inference-service] Waiting for CRD datascienceclusters.datasciencecluster.opendatahub.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD datascienceclusters.datasciencecluster.opendatahub.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io condition met [e2e-llm-inference-service] Applying all resources... [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/dscinitializations.dscinitialization.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencemodelrewrites.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceobjectives.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-llm-inference-service] serviceaccount/kserve-controller-manager serverside-applied [e2e-llm-inference-service] serviceaccount/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/kserve-leader-election-role serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/kserve-llmisvcconfig-read-access serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/llmisvc-leader-election-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-admin serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-edit serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-inferenceservice-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-manager-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-manager-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-metrics-reader-cluster-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-models-admin serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-models-edit serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-models-view serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-proxy-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-tls-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-view serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/openshift-ai-inferenceservice-image-volume-scc serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/kserve-leader-election-rolebinding serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/kserve-llmisvcconfig-read-access serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/llmisvc-leader-election-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-inferenceservice-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-llmisvc-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-manager-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-proxy-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-tls-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/llmisvc-manager-rolebinding serverside-applied [e2e-llm-inference-service] configmap/inferenceservice-config serverside-applied [e2e-llm-inference-service] configmap/kserve-parameters serverside-applied [e2e-llm-inference-service] secret/kserve-webhook-server-secret serverside-applied [e2e-llm-inference-service] secret/mlpipeline-s3-artifact serverside-applied [e2e-llm-inference-service] service/kserve-controller-manager-metrics-service serverside-applied [e2e-llm-inference-service] service/kserve-controller-manager-service serverside-applied [e2e-llm-inference-service] service/kserve-webhook-server-service serverside-applied [e2e-llm-inference-service] service/llmisvc-controller-manager-service serverside-applied [e2e-llm-inference-service] service/llmisvc-webhook-server-service serverside-applied [e2e-llm-inference-service] service/s3-service serverside-applied [e2e-llm-inference-service] deployment.apps/kserve-controller-manager serverside-applied [e2e-llm-inference-service] deployment.apps/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] deployment.apps/seaweedfs serverside-applied [e2e-llm-inference-service] networkpolicy.networking.k8s.io/kserve-controller-manager serverside-applied [e2e-llm-inference-service] securitycontextconstraints.security.openshift.io/openshift-ai-inferenceservice-image-volume-scc serverside-applied [e2e-llm-inference-service] securitycontextconstraints.security.openshift.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-llm-inference-service] clusterstoragecontainer.serving.kserve.io/default serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-router-route serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler-latency-predictor serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-tokenizer serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-tracing serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-worker-data-parallel serverside-applied [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/inferencegraph.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceserviceconfig.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/servingruntime.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/trainedmodel.serving.kserve.io serverside-applied [e2e-llm-inference-service] Waiting for llmisvc-controller-manager to be ready... [e2e-llm-inference-service] Waiting for pod -l "control-plane=llmisvc-controller-manager" in namespace "kserve" to be created... [e2e-llm-inference-service] Pod -l "control-plane=llmisvc-controller-manager" in namespace "kserve" found. [e2e-llm-inference-service] Current pods for -l "control-plane=llmisvc-controller-manager" in namespace "kserve": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] llmisvc-controller-manager-9547d8dfc-d5x59 0/1 Running 0 6s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "control-plane=llmisvc-controller-manager" in namespace "kserve" to become ready... [e2e-llm-inference-service] pod/llmisvc-controller-manager-9547d8dfc-d5x59 condition met [e2e-llm-inference-service] Pod(s) -l "control-plane=llmisvc-controller-manager" in namespace "kserve" are ready. [e2e-llm-inference-service] Re-Applying all resources... [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/dscinitializations.dscinitialization.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencemodelrewrites.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceobjectives.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-llm-inference-service] serviceaccount/kserve-controller-manager serverside-applied [e2e-llm-inference-service] serviceaccount/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/kserve-leader-election-role serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/kserve-llmisvcconfig-read-access serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/llmisvc-leader-election-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-admin serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-edit serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-inferenceservice-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-manager-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-manager-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-metrics-reader-cluster-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-models-admin serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-models-edit serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-models-view serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-proxy-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-tls-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-view serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/openshift-ai-inferenceservice-image-volume-scc serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/kserve-leader-election-rolebinding serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/kserve-llmisvcconfig-read-access serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/llmisvc-leader-election-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-inferenceservice-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-llmisvc-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-manager-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-proxy-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-tls-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/llmisvc-manager-rolebinding serverside-applied [e2e-llm-inference-service] configmap/inferenceservice-config serverside-applied [e2e-llm-inference-service] configmap/kserve-parameters serverside-applied [e2e-llm-inference-service] secret/kserve-webhook-server-secret serverside-applied [e2e-llm-inference-service] secret/mlpipeline-s3-artifact serverside-applied [e2e-llm-inference-service] service/kserve-controller-manager-metrics-service serverside-applied [e2e-llm-inference-service] service/kserve-controller-manager-service serverside-applied [e2e-llm-inference-service] service/kserve-webhook-server-service serverside-applied [e2e-llm-inference-service] service/llmisvc-controller-manager-service serverside-applied [e2e-llm-inference-service] service/llmisvc-webhook-server-service serverside-applied [e2e-llm-inference-service] service/s3-service serverside-applied [e2e-llm-inference-service] deployment.apps/kserve-controller-manager serverside-applied [e2e-llm-inference-service] deployment.apps/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] deployment.apps/seaweedfs serverside-applied [e2e-llm-inference-service] networkpolicy.networking.k8s.io/kserve-controller-manager serverside-applied [e2e-llm-inference-service] securitycontextconstraints.security.openshift.io/openshift-ai-inferenceservice-image-volume-scc serverside-applied [e2e-llm-inference-service] securitycontextconstraints.security.openshift.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-llm-inference-service] clusterstoragecontainer.serving.kserve.io/default serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-router-route serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler-latency-predictor serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-tokenizer serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-tracing serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-worker-data-parallel serverside-applied [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/inferencegraph.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceserviceconfig.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/servingruntime.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/trainedmodel.serving.kserve.io serverside-applied [e2e-llm-inference-service] Applying DSC/DSCI resources... [e2e-llm-inference-service] dscinitialization.dscinitialization.opendatahub.io/test-dsci created [e2e-llm-inference-service] datasciencecluster.datasciencecluster.opendatahub.io/test-dsc created [e2e-llm-inference-service] KServe manual installation complete [e2e-llm-inference-service] 🔧 Configuration: [e2e-llm-inference-service] KServe deployment: ❌ disabled [e2e-llm-inference-service] Kuadrant deployment: ✅ enabled [e2e-llm-inference-service] [e2e-llm-inference-service] Checking OpenShift server version...(4.21.26) [e2e-llm-inference-service] 🎯 Server version (4.21.26) is 4.19.9 or higher - continue with the script [e2e-llm-inference-service] ⏳ Installing cert-manager [e2e-llm-inference-service] namespace/cert-manager-operator created [e2e-llm-inference-service] operatorgroup.operators.coreos.com/openshift-cert-manager-operator created [e2e-llm-inference-service] subscription.operators.coreos.com/openshift-cert-manager-operator created [e2e-llm-inference-service] Waiting for openshift-cert-manager-operator CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (0/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (5/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (10/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (15/300) [e2e-llm-inference-service] CSV cert-manager-operator.v1.20.0 found, but not yet Succeeded (Phase: Installing). Waiting... (20/300) [e2e-llm-inference-service] CSV cert-manager-operator.v1.20.0 is ready (Phase: Succeeded). [e2e-llm-inference-service] Waiting for CRD certificates.cert-manager.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD certificates.cert-manager.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io condition met [e2e-llm-inference-service] ✅ Cert-manager installed [e2e-llm-inference-service] ⏳ Installing openshift-lws-operator [e2e-llm-inference-service] namespace/openshift-lws-operator created [e2e-llm-inference-service] operatorgroup.operators.coreos.com/leader-worker-set created [e2e-llm-inference-service] subscription.operators.coreos.com/leader-worker-set created [e2e-llm-inference-service] Waiting for leader-worker-set CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (0/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (5/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (10/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (15/300) [e2e-llm-inference-service] CSV leader-worker-set.v1.0.0 is ready (Phase: Succeeded). [e2e-llm-inference-service] Waiting for CRD leaderworkersetoperators.operator.openshift.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD leaderworkersetoperators.operator.openshift.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/leaderworkersetoperators.operator.openshift.io condition met [e2e-llm-inference-service] leaderworkersetoperator.operator.openshift.io/cluster created [e2e-llm-inference-service] ⏳ waiting for openshift-lws-operator to be ready.… [e2e-llm-inference-service] Waiting for pod -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" to be created... [e2e-llm-inference-service] Pod -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" found. [e2e-llm-inference-service] Current pods for -l "name=openshift-lws-operator" in namespace "openshift-lws-operator": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] openshift-lws-operator-fd8ccff4c-mzlfv 1/1 Running 0 13s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" to become ready... [e2e-llm-inference-service] pod/openshift-lws-operator-fd8ccff4c-mzlfv condition met [e2e-llm-inference-service] Pod(s) -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" are ready. [e2e-llm-inference-service] ✅ openshift-lws-operator installed [e2e-llm-inference-service] gatewayclass.gateway.networking.k8s.io/openshift-default created [e2e-llm-inference-service] Waiting for pod -l "app=istiod" in namespace "openshift-ingress" to be created... [e2e-llm-inference-service] Pod -l "app=istiod" in namespace "openshift-ingress" found. [e2e-llm-inference-service] Current pods for -l "app=istiod" in namespace "openshift-ingress": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] istiod-openshift-gateway-5677ccb49b-w6tmc 1/1 Running 0 6s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "app=istiod" in namespace "openshift-ingress" to become ready... [e2e-llm-inference-service] pod/istiod-openshift-gateway-5677ccb49b-w6tmc condition met [e2e-llm-inference-service] Pod(s) -l "app=istiod" in namespace "openshift-ingress" are ready. [e2e-llm-inference-service] ⏳ Creating a Gateway [e2e-llm-inference-service] Error from server (AlreadyExists): namespaces "openshift-ingress" already exists [e2e-llm-inference-service] ⏳ Creating gateway memory ConfigMap for parametersRef (2Gi) [e2e-llm-inference-service] configmap/gateway-proxy-config created [e2e-llm-inference-service] gateway.gateway.networking.k8s.io/openshift-ai-inference created [e2e-llm-inference-service] Waiting for pod -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" to be created... [e2e-llm-inference-service] Pod -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" found. [e2e-llm-inference-service] Current pods for -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf 0/1 Running 0 6s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" to become ready... [e2e-llm-inference-service] pod/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf condition met [e2e-llm-inference-service] Pod(s) -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" are ready. [e2e-llm-inference-service] ⏳ Installing RHCL(Kuadrant) operator [e2e-llm-inference-service] namespace/kuadrant-system created [e2e-llm-inference-service] subscription.operators.coreos.com/rhcl-operator created [e2e-llm-inference-service] operatorgroup.operators.coreos.com/kuadrant created [e2e-llm-inference-service] Waiting for rhcl-operator CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (0/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (5/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (10/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (15/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.2 found, but not yet Succeeded (Phase: Pending). Waiting... (20/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.2 found, but not yet Succeeded (Phase: Installing). Waiting... (25/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.2 found, but not yet Succeeded (Phase: Installing). Waiting... (30/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.2 found, but not yet Succeeded (Phase: Installing). Waiting... (35/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.2 is ready (Phase: Succeeded). [e2e-llm-inference-service] Waiting for CRD kuadrants.kuadrant.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD kuadrants.kuadrant.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/kuadrants.kuadrant.io condition met [e2e-llm-inference-service] Waiting for apiserver discovery /apis/kuadrant.io/v1beta1 to list kuadrants (timeout: 120s)… [e2e-llm-inference-service] Discovery for kuadrant.io/v1beta1 includes kuadrants. [e2e-llm-inference-service] ⏳ sleeping 30s after discovery (RESTMapper can trail discovery)… [e2e-llm-inference-service] kuadrant.kuadrant.io/kuadrant created [e2e-llm-inference-service] ⏳ waiting for Kuadrant Ready (attempt 1/2, timeout 5m)… [e2e-llm-inference-service] kuadrant.kuadrant.io/kuadrant condition met [e2e-llm-inference-service] Waiting for pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" to be created... [e2e-llm-inference-service] Pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" found. [e2e-llm-inference-service] Current pods for -l "control-plane=authorino-operator" in namespace "kuadrant-system": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] authorino-operator-56fb6f6856-tlmfp 1/1 Running 0 79s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" to become ready... [e2e-llm-inference-service] pod/authorino-operator-56fb6f6856-tlmfp condition met [e2e-llm-inference-service] Pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" are ready. [e2e-llm-inference-service] ⏳ waiting for authorino service to be created... [e2e-llm-inference-service] service/authorino-authorino-authorization condition met [e2e-llm-inference-service] service/authorino-authorino-authorization annotated [e2e-llm-inference-service] Warning: resource authorinos/authorino is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by oc apply. oc apply should only be used on resources created declaratively by either oc create --save-config or oc apply. The missing annotation will be patched automatically. [e2e-llm-inference-service] authorino.operator.authorino.kuadrant.io/authorino configured [e2e-llm-inference-service] Waiting for pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" to be created... [e2e-llm-inference-service] Pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" found. [e2e-llm-inference-service] Current pods for -l "control-plane=authorino-operator" in namespace "kuadrant-system": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] authorino-operator-56fb6f6856-tlmfp 1/1 Running 0 89s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" to become ready... [e2e-llm-inference-service] pod/authorino-operator-56fb6f6856-tlmfp condition met [e2e-llm-inference-service] Pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" are ready. [e2e-llm-inference-service] ✅ kuadrant(authorino) installed [e2e-llm-inference-service] Setting up Jaeger (Helm) for LLMISVC tracing e2e... [e2e-llm-inference-service] ⏳ Installing Jaeger All-in-One (Helm) into namespace observability [e2e-llm-inference-service] [INFO] Installing Helm v3.16.3 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed Helm v3.16.3 to /workspace/source/bin/helm [e2e-llm-inference-service] WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /credentials/cluster-tptt8-kubeconfig [e2e-llm-inference-service] WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /credentials/cluster-tptt8-kubeconfig [e2e-llm-inference-service] version.BuildInfo{Version:"v3.16.3", GitCommit:"cfd07493f46efc9debd9cc1b02a0961186df7fdf", GitTreeState:"clean", GoVersion:"go1.22.7"} [e2e-llm-inference-service] [INFO] Adding Jaeger Helm repository... [e2e-llm-inference-service] WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /credentials/cluster-tptt8-kubeconfig [e2e-llm-inference-service] WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /credentials/cluster-tptt8-kubeconfig [e2e-llm-inference-service] "jaegertracing" has been added to your repositories [e2e-llm-inference-service] [INFO] Installing Jaeger All-in-One 4.7.0... [e2e-llm-inference-service] WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /credentials/cluster-tptt8-kubeconfig [e2e-llm-inference-service] WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /credentials/cluster-tptt8-kubeconfig [e2e-llm-inference-service] NAME: jaeger [e2e-llm-inference-service] LAST DEPLOYED: Tue Jul 28 10:19:38 2026 [e2e-llm-inference-service] NAMESPACE: observability [e2e-llm-inference-service] STATUS: deployed [e2e-llm-inference-service] REVISION: 1 [e2e-llm-inference-service] TEST SUITE: None [e2e-llm-inference-service] NOTES: [e2e-llm-inference-service] ################################################################### [e2e-llm-inference-service] ### ⚠️ EXPERIMENTAL - NO STABILITY GUARANTEES ### [e2e-llm-inference-service] ### ### [e2e-llm-inference-service] ### This chart is under active development. ### [e2e-llm-inference-service] ### Breaking changes may occur in minor versions. ### [e2e-llm-inference-service] ### ### [e2e-llm-inference-service] ### See README.md for configuration details. ### [e2e-llm-inference-service] ################################################################### [e2e-llm-inference-service] [e2e-llm-inference-service] 🚀 Congratulations on successfully installing Jaeger v2.17.0 (Chart v4.7.0)! [e2e-llm-inference-service] [e2e-llm-inference-service] To access the query UI: [e2e-llm-inference-service] export POD_NAME=$(kubectl get pods --namespace observability -l "app.kubernetes.io/instance=jaeger,app.kubernetes.io/component=all-in-one" -o jsonpath="{.items[0].metadata.name}") [e2e-llm-inference-service] kubectl port-forward --namespace observability $POD_NAME 16686:16686 --address 0.0.0.0 & [e2e-llm-inference-service] Visit http://127.0.0.1:16686/ [e2e-llm-inference-service] [SUCCESS] Successfully installed Jaeger All-in-One via Helm [e2e-llm-inference-service] [INFO] Waiting for pods with label 'app.kubernetes.io/name=jaeger' in namespace 'observability' to be created... [e2e-llm-inference-service] [INFO] Found 1 pod(s) with label 'app.kubernetes.io/name=jaeger' [e2e-llm-inference-service] [INFO] Waiting for pods with label 'app.kubernetes.io/name=jaeger' in namespace 'observability' to be ready... [e2e-llm-inference-service] pod/jaeger-6bbd7d4c9d-x5v27 condition met [e2e-llm-inference-service] [SUCCESS] Pods with label 'app.kubernetes.io/name=jaeger' in namespace 'observability' are ready! [e2e-llm-inference-service] [SUCCESS] Jaeger is ready! [e2e-llm-inference-service] ⏳ Verifying Jaeger Service ports (OTLP 4317, Query 16686)... [e2e-llm-inference-service] ✅ Jaeger (Helm) ready — OTLP http://jaeger.observability.svc.cluster.local:4317 [e2e-llm-inference-service] Patching ingress domain... [e2e-llm-inference-service] configmap/inferenceservice-config patched [e2e-llm-inference-service] pod "kserve-controller-manager-6968749498-582ph" deleted [e2e-llm-inference-service] Waiting for kserve-controller-manager to be ready... [e2e-llm-inference-service] pod/kserve-controller-manager-6968749498-brk56 condition met [e2e-llm-inference-service] Installing ODH Model Controller manually... [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/accounts.nim.opendatahub.io created [e2e-llm-inference-service] serviceaccount/model-serving-api created [e2e-llm-inference-service] serviceaccount/odh-model-controller created [e2e-llm-inference-service] role.rbac.authorization.k8s.io/leader-election-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/account-editor-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/account-viewer-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-prometheus-k8s created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/metrics-reader created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/model-serving-api created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/odh-model-controller-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/proxy-role created [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/leader-election-rolebinding created [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/model-serving-api created [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/odh-model-controller-rolebinding-opendatahub created [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/proxy-rolebinding created [e2e-llm-inference-service] configmap/odh-model-controller-parameters created [e2e-llm-inference-service] service/model-serving-api created [e2e-llm-inference-service] service/odh-model-controller-metrics-service created [e2e-llm-inference-service] service/odh-model-controller-webhook-service created [e2e-llm-inference-service] deployment.apps/model-serving-api created [e2e-llm-inference-service] deployment.apps/odh-model-controller created [e2e-llm-inference-service] servicemonitor.monitoring.coreos.com/model-serving-api-metrics created [e2e-llm-inference-service] servicemonitor.monitoring.coreos.com/odh-model-controller-metrics-monitor created [e2e-llm-inference-service] template.template.openshift.io/autogluon-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/guardrails-detector-huggingface-serving-template created [e2e-llm-inference-service] template.template.openshift.io/kserve-ovms created [e2e-llm-inference-service] template.template.openshift.io/mlserver-cuda-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/mlserver-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-x86-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-x86-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-x86-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cuda-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cuda-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cuda-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-gaudi-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-gaudi-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-gaudi-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-multinode-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-multinode-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-multinode-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-rocm-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-rocm-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-rocm-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-ppc64le-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-ppc64le-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-ppc64le-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-s390x-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-s390x-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-s390x-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-x86-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-x86-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-x86-runtime-template-fast-2 created [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/mutating.odh-model-controller.opendatahub.io created [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/validating.odh-model-controller.opendatahub.io created [e2e-llm-inference-service] Waiting for deployment "odh-model-controller" rollout to finish: 0 of 1 updated replicas are available... [e2e-llm-inference-service] deployment "odh-model-controller" successfully rolled out [e2e-llm-inference-service] networkpolicy.networking.k8s.io/allow-all created [e2e-llm-inference-service] KServe setup complete (namespace: kserve) [e2e-llm-inference-service] Add testing models to SeaweedFS S3 storage ... [e2e-llm-inference-service] Waiting for SeaweedFS deployment to be ready... [e2e-llm-inference-service] deployment "seaweedfs" successfully rolled out [e2e-llm-inference-service] S3 init job not completed, re-creating... [e2e-llm-inference-service] job.batch/s3-init replaced [e2e-llm-inference-service] Waiting for S3 init job to complete... [e2e-llm-inference-service] job.batch/s3-init condition met [e2e-llm-inference-service] Prepare CI namespace and install ServingRuntimes [e2e-llm-inference-service] Setting up CI namespace: kserve-ci-e2e-test [e2e-llm-inference-service] Tearing down CI namespace: kserve-ci-e2e-test [e2e-llm-inference-service] Namespace kserve-ci-e2e-test does not exist, skipping deletion [e2e-llm-inference-service] CI namespace teardown complete [e2e-llm-inference-service] Creating namespace kserve-ci-e2e-test [e2e-llm-inference-service] namespace/kserve-ci-e2e-test created [e2e-llm-inference-service] Applying S3 artifact secret [e2e-llm-inference-service] secret/mlpipeline-s3-artifact created [e2e-llm-inference-service] Applying storage-config secret [e2e-llm-inference-service] secret/storage-config created [e2e-llm-inference-service] Applying SeaweedFS S3 credentials secret [e2e-llm-inference-service] secret/seaweedfs-s3-creds created [e2e-llm-inference-service] Linking seaweedfs-s3-creds to default service account [e2e-llm-inference-service] Creating odh-trusted-ca-bundle configmap [e2e-llm-inference-service] configmap/odh-trusted-ca-bundle created [e2e-llm-inference-service] Installing ServingRuntimes [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-autogluonserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-huggingfaceserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-huggingfaceserver-multinode created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-lgbserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-mlserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-paddleserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-pmmlserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-predictiveserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-sklearnserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-tensorflow-serving created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-torchserve created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-tritonserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-vllmserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-xgbserver created [e2e-llm-inference-service] CI namespace setup complete [e2e-llm-inference-service] Setup complete [e2e-llm-inference-service] === E2E cluster / operator summary === [e2e-llm-inference-service] Client Version: 4.20.11 [e2e-llm-inference-service] Kustomize Version: v5.6.0 [e2e-llm-inference-service] Server Version: 4.21.26 [e2e-llm-inference-service] Kubernetes Version: v1.34.9 [e2e-llm-inference-service] ClusterVersion desired: 4.21.26 [e2e-llm-inference-service] ClusterVersion history (latest): 4.21.26 (Completed) [e2e-llm-inference-service] CSVs in kuadrant-system: [e2e-llm-inference-service] authorino-operator.v1.4.2 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.0 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.1 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.1 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.2 Succeeded [e2e-llm-inference-service] CSVs in openshift-keda: [e2e-llm-inference-service] authorino-operator.v1.4.2 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.0 Succeeded [e2e-llm-inference-service] custom-metrics-autoscaler.v2.19.0-2 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.1 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.1 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.2 Succeeded [e2e-llm-inference-service] CSVs in cert-manager-operator: [e2e-llm-inference-service] authorino-operator.v1.4.2 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.0 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.1 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.1 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.2 Succeeded [e2e-llm-inference-service] CSVs in openshift-lws-operator: [e2e-llm-inference-service] authorino-operator.v1.4.2 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.0 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.1 Succeeded [e2e-llm-inference-service] leader-worker-set.v1.0.0 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.1 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.2 Succeeded [e2e-llm-inference-service] CSVs in openshift-operators (ODH / shared operators, filtered): [e2e-llm-inference-service] authorino-operator.v1.4.2 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.1 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.1 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.2 Succeeded [e2e-llm-inference-service] Kuadrant / Authorino (diagnostics): [e2e-llm-inference-service] CRD kuadrants.kuadrant.io versions: v1beta1 served=true storage=true [e2e-llm-inference-service] Subscriptions in kuadrant-system: [e2e-llm-inference-service] authorino-operator-stable-redhat-operators-openshift-marketplace stable redhat-operators authorino-operator.v1.4.2 [e2e-llm-inference-service] dns-operator-stable-redhat-operators-openshift-marketplace stable redhat-operators dns-operator.v1.4.1 [e2e-llm-inference-service] limitador-operator-stable-redhat-operators-openshift-marketplace stable redhat-operators limitador-operator.v1.4.1 [e2e-llm-inference-service] rhcl-operator stable redhat-operators rhcl-operator.v1.4.2 [e2e-llm-inference-service] Kuadrant CR conditions (kuadrant/kuadrant-system): [e2e-llm-inference-service] Ready=True (Ready) [e2e-llm-inference-service] KServe deployments in kserve: [e2e-llm-inference-service] kserve-controller-manager: ready=1 image=quay.io/opendatahub/kserve-controller@sha256:a00b96ab2f9202facf950a577d55a24338ee980c7c01fced43402b87d97964be [e2e-llm-inference-service] imageID: quay.io/opendatahub/kserve-controller@sha256:121f4d8b1c0c993e610878afa80cb66895ca258f727dad8ed8dfdc6b1d4cc1c3 [e2e-llm-inference-service] odh-model-controller: ready=1 image=quay.io/opendatahub/odh-model-controller:odh-incubating [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-model-controller@sha256:b5c835d640b650f164ba7f36d7008760c88e2974efa65853ee167323c441580c [e2e-llm-inference-service] llmisvc-controller-manager: ready=1 image=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:b2c041a1b3fc0af41ba7db1aab2006d510e30f800ca4162b46f8fde3a4726d62 [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:21dd391c4037442599f3ab2760fb6862b2f774734e7a5a840635c9868729a485 [e2e-llm-inference-service] === End E2E cluster / operator summary === [e2e-llm-inference-service] /workspace/source [e2e-llm-inference-service] CA certificate extracted [e2e-llm-inference-service] REQUESTS_CA_BUNDLE=/tmp/ca.crt [e2e-llm-inference-service] Run E2E tests: llminferenceservice and cluster_cpu and not autoscaling and not tracing [e2e-llm-inference-service] Starting E2E functional tests ... [e2e-llm-inference-service] Parallelism requested for pytest is 2 [e2e-llm-inference-service] ============================= test session starts ============================== [e2e-llm-inference-service] platform linux -- Python 3.11.13, pytest-7.4.4, pluggy-1.5.0 -- /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] cachedir: .pytest_cache [e2e-llm-inference-service] metadata: {'Python': '3.11.13', 'Platform': 'Linux-5.14.0-570.122.1.el9_6.x86_64-x86_64-with-glibc2.34', 'Packages': {'pytest': '7.4.4', 'pluggy': '1.5.0'}, 'Plugins': {'anyio': '4.9.0', 'asyncio': '0.23.8', 'xdist': '3.6.1', 'cov': '5.0.0', 'httpx': '0.30.0', 'json-report': '1.5.0', 'metadata': '3.1.1'}, 'PLATFORM': 'el9'} [e2e-llm-inference-service] rootdir: /workspace/source/test/e2e [e2e-llm-inference-service] configfile: pytest.ini [e2e-llm-inference-service] plugins: anyio-4.9.0, asyncio-0.23.8, xdist-3.6.1, cov-5.0.0, httpx-0.30.0, json-report-1.5.0, metadata-3.1.1 [e2e-llm-inference-service] asyncio: mode=Mode.STRICT [e2e-llm-inference-service] created: 2/2 workers [e2e-llm-inference-service] 2 workers [73 items] [e2e-llm-inference-service] [e2e-llm-inference-service] scheduling tests via WorkStealingScheduling [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-utilization-detector] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] 2026-07-28 10:22:05.433 6508 kserve INFO [conftest.py:configure_logger():40] Logger configured [e2e-llm-inference-service] 2026-07-28 10:22:05.434 6505 kserve INFO [conftest.py:configure_logger():40] Logger configured [e2e-llm-inference-service] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-utilization-detector] [e2e-llm-inference-service] llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-concurrency-detector] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-concurrency-detector] [e2e-llm-inference-service] llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-with-section-name] 2026-07-28 10:24:37.505 6505 kserve.trace Checking Gateway router-gateway-1 in namespace e2e-test-gateway-section-name-propagation-49b5edd9 [e2e-llm-inference-service] 2026-07-28 10:24:37.505 6505 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-1 in namespace e2e-test-gateway-section-name-propagation-49b5edd9 [e2e-llm-inference-service] 2026-07-28 10:24:37.534 6505 kserve.trace Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-28 10:24:37.534 6505 kserve.trace INFO [gw_api.py:create_or_update_gateway():62] Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-28 10:24:37.543 6505 kserve.trace ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-28 10:24:37.543 6505 kserve.trace INFO [gw_api.py:create_or_update_gateway():70] ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-with-section-name] [e2e-llm-inference-service] llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-without-section-name] 2026-07-28 10:24:51.664 6505 kserve.trace Checking Gateway router-gateway-1 in namespace e2e-test-gateway-section-name-propagation-94799d44 [e2e-llm-inference-service] 2026-07-28 10:24:51.664 6505 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-1 in namespace e2e-test-gateway-section-name-propagation-94799d44 [e2e-llm-inference-service] 2026-07-28 10:24:51.696 6505 kserve.trace Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-28 10:24:51.696 6505 kserve.trace INFO [gw_api.py:create_or_update_gateway():62] Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-28 10:24:51.708 6505 kserve.trace ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-28 10:24:51.708 6505 kserve.trace INFO [gw_api.py:create_or_update_gateway():70] ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-without-section-name] [e2e-llm-inference-service] llmisvc/test_llm_auth.py::test_llm_auth_enabled_requires_token[cluster_cpu-cluster_single_node-auth-enabled-default] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_auth.py::test_llm_auth_enabled_requires_token[cluster_cpu-cluster_single_node-auth-enabled-default] [e2e-llm-inference-service] llmisvc/test_llm_auth.py::test_llm_auth_invalid_token_rejected[cluster_cpu-cluster_single_node-auth-invalid-token] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-inline-config-workload-llmd-simulator] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-inline-config-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-qwen2.5-0.5b] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_auth.py::test_llm_auth_invalid_token_rejected[cluster_cpu-cluster_single_node-auth-invalid-token] [e2e-llm-inference-service] llmisvc/test_llm_auth.py::test_llm_auth_disabled_no_token_required[cluster_cpu-cluster_single_node-auth-disabled] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-qwen2.5-0.5b] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-configmap-ref-workload-llmd-simulator] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-configmap-ref-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-replicas-workload-llmd-simulator] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-replicas-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-custom-template-workload-llmd-simulator] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_auth.py::test_llm_auth_disabled_no_token_required[cluster_cpu-cluster_single_node-auth-disabled] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_hpa_deployment[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-custom-template-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-nonzero-threshold-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-nonzero-threshold-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-tokenizer-kvcache-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-tokenizer-kvcache-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator0] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator1] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator2] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator2] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf0] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] [e2e-llm-inference-service] [gw0] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_hpa_deployment[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_keda_deployment[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-keda] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-pvc] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-pvc] [e2e-llm-inference-service] [gw0] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_keda_deployment[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-keda] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_hpa_lws[cluster_cpu-cluster_multi_node-router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] [gw1] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-pvc] [e2e-llm-inference-service] [gw0] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_hpa_lws[cluster_cpu-cluster_multi_node-router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_keda_lws[cluster_cpu-cluster_multi_node-router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-keda] [e2e-llm-inference-service] [gw1] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service_config_deletion.py::test_config_finalizer_added [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_config_deletion.py::test_config_finalizer_added [e2e-llm-inference-service] llmisvc/test_llm_inference_service_config_deletion.py::test_config_deletion_blocked_when_referenced [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_config_deletion.py::test_config_deletion_blocked_when_referenced [e2e-llm-inference-service] llmisvc/test_llm_inference_service_config_deletion.py::test_config_deletion_allowed_when_unreferenced [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_config_deletion.py::test_config_deletion_allowed_when_unreferenced [e2e-llm-inference-service] llmisvc/test_llm_inference_service_config_deletion.py::test_config_deletion_unblocked_after_service_deleted [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_config_deletion.py::test_config_deletion_unblocked_after_service_deleted [e2e-llm-inference-service] llmisvc/test_llm_inference_service_config_deletion.py::test_well_known_config_deletion_prevented_by_webhook [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_config_deletion.py::test_well_known_config_deletion_prevented_by_webhook [e2e-llm-inference-service] llmisvc/test_llm_inference_service_config_deletion.py::test_well_known_config_deletion_blocked_by_implicit_reference [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_config_deletion.py::test_well_known_config_deletion_blocked_by_implicit_reference [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha1_to_v1alpha2_conversion [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha1_to_v1alpha2_conversion [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha2_to_v1alpha1_conversion [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha2_to_v1alpha1_conversion [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_criticality_preservation_via_annotations [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_criticality_preservation_via_annotations [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_lora_criticality_preservation [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_lora_criticality_preservation [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_round_trip_conversion_preserves_fields [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_round_trip_conversion_preserves_fields [e2e-llm-inference-service] llmisvc/test_llm_inference_service_stop.py::test_llm_stop_feature[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_stop.py::test_llm_stop_feature[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-single-lora-adapter-hf] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-single-lora-adapter-hf] [e2e-llm-inference-service] llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-multiple-lora-adapters] [e2e-llm-inference-service] [gw0] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_keda_lws[cluster_cpu-cluster_multi_node-router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-keda] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_cleanup_hpa[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-multiple-lora-adapters] [e2e-llm-inference-service] llmisvc/test_llm_tls.py::test_llm_tls_resources[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_tls.py::test_llm_tls_resources[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_prestop_hook.py::test_prestop_hook[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_prestop_hook.py::test_prestop_hook[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_rolling_upgrade.py::test_rolling_upgrade_coordination[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_rolling_upgrade.py::test_rolling_upgrade_coordination[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_storage_version_migration.py::TestStorageVersionMigration::test_storage_version_migration_after_simulated_upgrade [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_storage_version_migration.py::TestStorageVersionMigration::test_storage_version_migration_after_simulated_upgrade [e2e-llm-inference-service] [gw0] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_cleanup_hpa[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] [e2e-llm-inference-service] =================================== FAILURES =================================== [e2e-llm-inference-service] _ test_llm_autoscaling_hpa_deployment[router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-hpa] _ [e2e-llm-inference-service] [gw0] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-hpa'], prom... {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.autoscaling_hpa [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator-no-replicas", [e2e-llm-inference-service] "prometheus-scrape", [e2e-llm-inference-service] "scaling-hpa", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="autoscale-hpa-deploy", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_autoscaling_hpa_deployment(test_case: TestCase): [e2e-llm-inference-service] """HPA + Deployment: HPA exists with WVA annotations; pods scale up under load.""" [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] kserve_client = _new_kserve_client() [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] ns = test_case.namespace [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > _create_and_wait(kserve_client, test_case) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:542: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-hpa'], prom... {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] def _create_and_wait(kserve_client, test_case): [e2e-llm-inference-service] """Create LLMISVC and wait for it to be ready.""" [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:482: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...e-hpa-4c186bcf'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-07-28T10:34:25.545003', start_time = 1785234865.5452309 [e2e-llm-inference-service] duration = 900.7892689704895, timestamp_end = '2026-07-28T10:49:26.334507' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....toscale-hpa-4c186bcf'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1376: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7ff792dd04a0> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1387: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T10:34:55Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T10:34:55Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T10:34:55Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T10:34:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T10:35:11Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T10:35:11Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T10:34:55Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T10:35:11Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T10:34:55Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1371: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:82 Created test namespace e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret seaweedfs-s3-creds from kserve-ci-e2e-test to e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret storage-config from kserve-ci-e2e-test to e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:178 Patched default SA in e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 with secret seaweedfs-s3-creds [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:162 ConfigMap odh-kserve-custom-ca-bundle already exists in e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:159 Copied ConfigMap odh-trusted-ca-bundle from kserve-ci-e2e-test to e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig router-managed-autoscale-hpa-de-ec1dce8b in namespace e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig router-managed-autoscale-hpa-de-ec1dce8b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig router-managed-autoscale-hpa-de-ec1dce8b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig workload-llmd-simulator-no-repl-38916baa in namespace e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig workload-llmd-simulator-no-repl-38916baa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig workload-llmd-simulator-no-repl-38916baa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig prometheus-scrape-autoscale-hpa-4c186bcf in namespace e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig prometheus-scrape-autoscale-hpa-4c186bcf [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig prometheus-scrape-autoscale-hpa-4c186bcf [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig scaling-hpa-autoscale-hpa-deplo-347a3180 in namespace e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig scaling-hpa-autoscale-hpa-deplo-347a3180 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig scaling-hpa-autoscale-hpa-deplo-347a3180 [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_autoscaling_hpa_deployment] [2026-07-28T10:34:25.372784] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-hpa'], prompt='KServe is a', service_name='autoscale-hpa-deploy', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, namespace='e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4', before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-deploy', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-de-ec1dce8b'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-38916baa'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-4c186bcf'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-28T10:34:25.385069] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-deploy', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-de-ec1dce8b'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-38916baa'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-4c186bcf'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-28T10:34:25.544854] end - ✅ in 0.160s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-28T10:34:25.545003] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-deploy', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-de-ec1dce8b'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-38916baa'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-4c186bcf'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'RouterReady', 'WorkloadsReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T10:34:39Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T10:34:39Z', 'message': 'Inference Pool e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T10:34:39Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T10:34:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T10:34:39Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T10:34:39Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T10:34:39Z', 'message': 'HPA conditions not yet available', 'reason': 'HPAProgressing', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T10:34:39Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T10:34:39Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'RouterReady', 'WorkloadsReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T10:34:55Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T10:34:55Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T10:34:55Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T10:34:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T10:34:55Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T10:34:55Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T10:34:55Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T10:34:55Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T10:34:55Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T10:34:55Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T10:34:55Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T10:34:55Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T10:34:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T10:35:11Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T10:35:11Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T10:34:55Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T10:35:11Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T10:34:55Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 3: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-28T10:22:24Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 4: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-28T10:22:24Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 5: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-28T10:22:24Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1391 Timed out waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T10:34:55Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T10:34:55Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T10:34:55Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T10:34:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T10:35:11Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T10:35:11Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T10:34:55Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T10:35:11Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T10:34:55Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-07-28T10:49:26.334507] end - ❌ 900.789s: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T10:34:55Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T10:34:55Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T10:34:55Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T10:34:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T10:35:11Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T10:35:11Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T10:34:55Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T10:35:11Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T10:34:55Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [delete_llmisvc] [2026-07-28T10:49:26.334702] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-deploy', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-de-ec1dce8b'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-38916baa'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-4c186bcf'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 2 pod(s) for autoscale-hpa-deploy still terminating: ['autoscale-hpa-deploy-kserve-66746bd8b9-8gpjm', 'autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-986gr'] [e2e-llm-inference-service] assert not ['autoscale-hpa-deploy-kserve-66746bd8b9-8gpjm', 'autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-986gr'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 1 pod(s) for autoscale-hpa-deploy still terminating: ['autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-986gr'] [e2e-llm-inference-service] assert not ['autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-986gr'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [delete_llmisvc] [2026-07-28T10:49:46.625460] end - ✅ in 20.290s [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_autoscaling_hpa_deployment] [2026-07-28T10:49:46.625710] end - ❌ 921.253s: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T10:34:55Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T10:34:55Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T10:34:55Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T10:34:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T10:35:11Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T10:35:11Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T10:34:55Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T10:35:11Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T10:34:55Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ---------------------------- Captured log teardown ----------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.conftest:conftest.py:168 Skipping deletion of namespace e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 (SKIP_DELETION_ON_FAILURE) [e2e-llm-inference-service] _ test_llm_autoscaling_keda_deployment[router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-keda] _ [e2e-llm-inference-service] [gw0] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-keda'], pro... {'name': 'scaling-keda-autoscale-keda-dep-1ac84077'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.autoscaling_keda [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator-no-replicas", [e2e-llm-inference-service] "prometheus-scrape", [e2e-llm-inference-service] "scaling-keda", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="autoscale-keda-deploy", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_autoscaling_keda_deployment(test_case: TestCase): [e2e-llm-inference-service] """KEDA + Deployment: ScaledObject exists with WVA annotations; no HPA; pods scale up under load.""" [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] kserve_client = _new_kserve_client() [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] ns = test_case.namespace [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > _create_and_wait(kserve_client, test_case) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:606: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-keda'], pro... {'name': 'scaling-keda-autoscale-keda-dep-1ac84077'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] def _create_and_wait(kserve_client, test_case): [e2e-llm-inference-service] """Create LLMISVC and wait for it to be ready.""" [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:482: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...e-ked-101f2a9d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-dep-1ac84077'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-07-28T10:49:47.833705', start_time = 1785235787.8340697 [e2e-llm-inference-service] duration = 900.8916921615601, timestamp_end = '2026-07-28T11:04:48.725777' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....toscale-ked-101f2a9d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-dep-1ac84077'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1376: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7ff792df44a0> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1387: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T10:50:30Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T10:50:30Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T10:50:30Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T10:50:10Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T10:50:41Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T10:50:41Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T10:50:30Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T10:50:41Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T10:50:30Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1371: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:82 Created test namespace e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret seaweedfs-s3-creds from kserve-ci-e2e-test to e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret storage-config from kserve-ci-e2e-test to e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:178 Patched default SA in e2e-test-llm-autoscaling-keda-deployment-b2150d0b with secret seaweedfs-s3-creds [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:162 ConfigMap odh-kserve-custom-ca-bundle already exists in e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:159 Copied ConfigMap odh-trusted-ca-bundle from kserve-ci-e2e-test to e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig router-managed-autoscale-keda-d-27e06c40 in namespace e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig router-managed-autoscale-keda-d-27e06c40 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig router-managed-autoscale-keda-d-27e06c40 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig workload-llmd-simulator-no-repl-da49e827 in namespace e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig workload-llmd-simulator-no-repl-da49e827 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig workload-llmd-simulator-no-repl-da49e827 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig prometheus-scrape-autoscale-ked-101f2a9d in namespace e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig prometheus-scrape-autoscale-ked-101f2a9d [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig prometheus-scrape-autoscale-ked-101f2a9d [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig scaling-keda-autoscale-keda-dep-1ac84077 in namespace e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig scaling-keda-autoscale-keda-dep-1ac84077 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig scaling-keda-autoscale-keda-dep-1ac84077 [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_autoscaling_keda_deployment] [2026-07-28T10:49:47.668737] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-keda'], prompt='KServe is a', service_name='autoscale-keda-deploy', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, namespace='e2e-test-llm-autoscaling-keda-deployment-b2150d0b', before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-keda-deploy', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-keda-deployment-b2150d0b', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-keda-d-27e06c40'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-da49e827'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-ked-101f2a9d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-dep-1ac84077'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-28T10:49:47.680968] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-keda-deploy', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-keda-deployment-b2150d0b', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-keda-d-27e06c40'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-da49e827'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-ked-101f2a9d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-dep-1ac84077'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-28T10:49:47.833499] end - ✅ in 0.152s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-28T10:49:47.833705] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-keda-deploy', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-keda-deployment-b2150d0b', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-keda-d-27e06c40'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-da49e827'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-ked-101f2a9d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-dep-1ac84077'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'RouterReady', 'WorkloadsReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T10:50:10Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T10:50:10Z', 'message': 'Inference Pool e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T10:50:10Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T10:50:10Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T10:50:10Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T10:50:10Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T10:50:10Z', 'message': 'ScaledObject not yet visible in cache', 'reason': 'ScaledObjectProgressing', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T10:50:10Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T10:50:10Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'RouterReady', 'WorkloadsReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T10:50:30Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T10:50:30Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T10:50:30Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T10:50:10Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T10:50:30Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T10:50:30Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T10:50:30Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T10:50:30Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T10:50:30Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T10:50:30Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T10:50:30Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T10:50:30Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T10:50:10Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T10:50:41Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T10:50:41Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T10:50:30Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T10:50:41Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T10:50:30Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 6: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-28T10:22:24Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 7: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-28T10:22:24Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 8: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-28T10:22:24Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1391 Timed out waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T10:50:30Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T10:50:30Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T10:50:30Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T10:50:10Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T10:50:41Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T10:50:41Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T10:50:30Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T10:50:41Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T10:50:30Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-07-28T11:04:48.725777] end - ❌ 900.892s: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T10:50:30Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T10:50:30Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T10:50:30Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T10:50:10Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T10:50:41Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T10:50:41Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T10:50:30Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T10:50:41Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T10:50:30Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [delete_llmisvc] [2026-07-28T11:04:48.725950] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-keda-deploy', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-keda-deployment-b2150d0b', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-keda-d-27e06c40'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-da49e827'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-ked-101f2a9d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-dep-1ac84077'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 2 pod(s) for autoscale-keda-deploy still terminating: ['autoscale-keda-deploy-kserve-77fc6d7cf9-22kqh', 'autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-rhmxf'] [e2e-llm-inference-service] assert not ['autoscale-keda-deploy-kserve-77fc6d7cf9-22kqh', 'autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-rhmxf'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 1 pod(s) for autoscale-keda-deploy still terminating: ['autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-rhmxf'] [e2e-llm-inference-service] assert not ['autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-rhmxf'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [delete_llmisvc] [2026-07-28T11:05:24.171499] end - ✅ in 35.445s [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_autoscaling_keda_deployment] [2026-07-28T11:05:24.171667] end - ❌ 936.503s: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T10:50:30Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T10:50:30Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T10:50:30Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T10:50:10Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T10:50:41Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T10:50:41Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T10:50:30Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T10:50:41Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T10:50:30Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ---------------------------- Captured log teardown ----------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.conftest:conftest.py:168 Skipping deletion of namespace e2e-test-llm-autoscaling-keda-deployment-b2150d0b (SKIP_DELETION_ON_FAILURE) [e2e-llm-inference-service] _____ test_llm_inference_service[router-managed-workload-pd-cpu-model-pvc] _____ [e2e-llm-inference-service] [gw1] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-pd-cpu', 'model-pvc'], prompt='KServe is a', service_name='llmisvc-mod... {'name': 'model-pvc-llmisvc-model-pvc-rou-49c1f027'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-gateway-ref", [e2e-llm-inference-service] "router-with-managed-route", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] expected_gateway="router-gateway-1", [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda tc: create_router_resources( [e2e-llm-inference-service] gateways=[ [e2e-llm-inference-service] make_router_gateway( [e2e-llm-inference-service] "router-gateway-1", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="custom-route-timeout-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="router-with-refs-test", [e2e-llm-inference-service] expected_gateway="router-gateway-1", [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda tc: create_router_resources( [e2e-llm-inference-service] gateways=[ [e2e-llm-inference-service] make_router_gateway( [e2e-llm-inference-service] "router-gateway-1", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] routes=[ [e2e-llm-inference-service] make_router_main_route( [e2e-llm-inference-service] "router-route-1", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] "router-gateway-1", [e2e-llm-inference-service] "router-with-refs-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] make_router_health_route( [e2e-llm-inference-service] "router-route-2", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] "router-gateway-1", [e2e-llm-inference-service] "router-with-refs-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=["router-managed", "workload-pd-cpu", "model-fb-opt-125m"], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="custom-route-timeout-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="router-with-refs-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] expected_gateway="router-gateway-2", [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda tc: create_router_resources( [e2e-llm-inference-service] gateways=[ [e2e-llm-inference-service] make_router_gateway( [e2e-llm-inference-service] "router-gateway-2", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] routes=[ [e2e-llm-inference-service] make_router_main_route( [e2e-llm-inference-service] "router-route-3", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] "router-gateway-2", [e2e-llm-inference-service] "router-with-refs-pd-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] make_router_health_route( [e2e-llm-inference-service] "router-route-4", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] "router-gateway-2", [e2e-llm-inference-service] "router-with-refs-pd-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-dp-ep-gpu", [e2e-llm-inference-service] "workload-dp-ep-prefill-gpu", [e2e-llm-inference-service] "model-deepseek-v2-lite", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="Delve into the multifaceted implications of a fully disaggregated cloud architecture, specifically " [e2e-llm-inference-service] "where the compute plane (P) and the data plane (D) are independently deployed and managed for a " [e2e-llm-inference-service] "geographically distributed, high-throughput, low-latency microservices ecosystem. Beyond the " [e2e-llm-inference-service] "fundamental challenges of network latency and data consistency, elaborate on the advanced " [e2e-llm-inference-service] "considerations and trade-offs inherent in such a setup: 1. Network Architecture and Protocols: " [e2e-llm-inference-service] "How would the network fabric and underlying protocols (e.g., RDMA, custom transport layers) need to " [e2e-llm-inference-service] "evolve to support optimal performance and minimize inter-plane communication overhead, especially for " [e2e-llm-inference-service] "synchronous operations? Discuss the role of network programmability (e.g., SDN, P4) in dynamically " [e2e-llm-inference-service] "optimizing routing and traffic flow between P and D. 2. Advanced Data Consistency and Durability: " [e2e-llm-inference-service] "Explore sophisticated data consistency models (e.g., causal consistency, strong eventual consistency) " [e2e-llm-inference-service] "and their applicability in balancing performance and data integrity across a globally distributed data plane. " [e2e-llm-inference-service] "Detail strategies for ensuring data durability and fault tolerance, including multi-region replication, " [e2e-llm-inference-service] "intelligent partitioning, and recovery mechanisms in the event of partial or full plane failures. " [e2e-llm-inference-service] "3. Dynamic Resource Orchestration and Cost Optimization: Analyze how an orchestration layer would intelligently " [e2e-llm-inference-service] "manage the independent scaling of compute (P) and data (D) resources, considering fluctuating workloads, " [e2e-llm-inference-service] "cost efficiency, and performance targets (e.g., using predictive analytics for resource provisioning). " [e2e-llm-inference-service] "Discuss mechanisms for dynamically reallocating compute nodes to different data partitions based on " [e2e-llm-inference-service] "workload patterns and data locality, potentially involving live migration strategies. " [e2e-llm-inference-service] "4. Security and Compliance in a Distributed Landscape: Address the enhanced security perimeter " [e2e-llm-inference-service] "challenges, including securing communication channels between P and D (encryption in transit, mutual TLS), " [e2e-llm-inference-service] "fine-grained access control to data at rest and in motion, and identity management across disaggregated " [e2e-llm-inference-service] "components. Discuss how such an architecture impacts compliance with regulatory frameworks (e.g., GDPR, HIPAA) " [e2e-llm-inference-service] "concerning data sovereignty, privacy, and auditability. 5. Operational Complexity and Observability: " [e2e-llm-inference-service] "Examine the increased complexity in monitoring, logging, and tracing across highly decoupled compute and " [e2e-llm-inference-service] "data planes. What specialized tooling and practices (e.g., distributed tracing with OpenTelemetry, advanced AIOps) " [e2e-llm-inference-service] "would be essential? How would incident response and troubleshooting differ in this disaggregated environment " [e2e-llm-inference-service] "compared to traditional integrated systems? Consider the challenges of pinpointing root causes across " [e2e-llm-inference-service] "independent failures. 6. Real-world Applicability and Future Trends: Identify specific industries " [e2e-llm-inference-service] "or use cases (e.g., high-frequency trading, IoT edge processing, large language model inference) " [e2e-llm-inference-service] "where the benefits of P/D disaggregation would strongly outweigh its complexities. " [e2e-llm-inference-service] "Conclude by speculating on emerging technologies or paradigms (e.g., serverless compute functions " [e2e-llm-inference-service] "directly interacting with object storage, in-memory disaggregation) that could further drive or " [e2e-llm-inference-service] "transform P/D disaggregation in cloud computing.", [e2e-llm-inference-service] max_tokens=2000, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_gpu, [e2e-llm-inference-service] pytest.mark.cluster_nvidia, [e2e-llm-inference-service] pytest.mark.cluster_nvidia_roce, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-no-scheduler", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.no_scheduler, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="This test simulates DP+EP that can run on CPU, the idea is to test the LWS-based deployment, " [e2e-llm-inference-service] "but without the resources requirements for DP+EP (GPUs and ROCe/IB).", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_multi_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler config tests [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-inline-config-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Chat completions endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] model_name="Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-configmap-ref", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-configmap-ref-test", [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda tc: create_scheduler_configmap(namespace=tc.namespace) [e2e-llm-inference-service] ], [e2e-llm-inference-service] after_test=[ [e2e-llm-inference-service] lambda tc: delete_scheduler_configmap(namespace=tc.namespace) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-replicas", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-ha-replicas-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-custom-template", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-custom-template-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler v0.6 → v0.7 migration tests. [e2e-llm-inference-service] # Deploy v0.6-style configs and verify the controller migrates them [e2e-llm-inference-service] # so the v0.7 scheduler boots successfully. [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-v06-pd-config-migration", [e2e-llm-inference-service] "workload-llmd-simulator-pd", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-v06-pd-migration-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-v06-nonzero-threshold-migration", [e2e-llm-inference-service] "workload-llmd-simulator-pd", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-v06-threshold-migration-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Standalone tokenizer — clean path: token-producer in inline config [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-tokenizer-kvcache", [e2e-llm-inference-service] "workload-llmd-simulator-kvcache", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="tokenizer-clean-path-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Standalone tokenizer — migration path: legacy precise-prefix-cache-scorer [e2e-llm-inference-service] # triggers auto-provisioned tokenizer without explicit tokenizer:{} field [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-precise-prefix-cache-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator-kvcache", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="tokenizer-migration-path-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Models endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="data"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/chat/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — LoRA adapter [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] model_name="publishers/{namespace}/models/lora-adapter-1", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "publishers/{namespace}/models/lora-adapter-1" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/lora-adapter-1", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/models (base + LoRA) [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=assert_models_contains( [e2e-llm-inference-service] "facebook/opt-125m", [e2e-llm-inference-service] "publishers/{namespace}/models/facebook/opt-125m", [e2e-llm-inference-service] "lora-adapter-1", [e2e-llm-inference-service] "publishers/{namespace}/models/lora-adapter-1", [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # PVC storage tests -- validate direct PVC volume mount with real vLLM serving [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] before_test=[lambda tc: ensure_pvc_with_model(namespace=tc.namespace)], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] before_test=[lambda tc: ensure_pvc_with_model(namespace=tc.namespace)], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] before_test=[lambda tc: ensure_pvc_with_model(namespace=tc.namespace)], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_multi_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_inference_service(test_case: TestCase): # noqa: F811 [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = KServeClient( [e2e-llm-inference-service] config_file=os.environ.get("KUBECONFIG", "~/.kube/config"), [e2e-llm-inference-service] client_configuration=client.Configuration(), [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] prefix = test_case.log_prefix [e2e-llm-inference-service] [e2e-llm-inference-service] test_failed = False [e2e-llm-inference-service] try: [e2e-llm-inference-service] print(f"{prefix} Creating LLMInferenceService {service_name}") [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] print(f"{prefix} Waiting for LLMInferenceService {service_name} to be ready") [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:866: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...del-p-9d807ba3'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-49c1f027'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-07-28T10:55:06.253418', start_time = 1785236106.2538621 [e2e-llm-inference-service] duration = 900.3814918994904, timestamp_end = '2026-07-28T11:10:06.635359' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....svc-model-p-9d807ba3'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-49c1f027'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1376: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7f7bf864d8a0> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1387: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T10:55:30Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T10:55:30Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T10:57:33Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T10:55:30Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-28T10:55:21Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T10:55:30Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T10:55:52Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T10:55:52Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T10:55:30Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1371: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:82 Created test namespace e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret seaweedfs-s3-creds from kserve-ci-e2e-test to e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret storage-config from kserve-ci-e2e-test to e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:178 Patched default SA in e2e-test-llm-inference-service-48639af5 with secret seaweedfs-s3-creds [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:162 ConfigMap odh-kserve-custom-ca-bundle already exists in e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:159 Copied ConfigMap odh-trusted-ca-bundle from kserve-ci-e2e-test to e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1959 Created PVC e2e-pvc-model-storage in namespace e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2099 Created model download Job e2e-pvc-model-download in namespace e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2124 Model download Job e2e-pvc-model-download completed successfully [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig router-managed-llmisvc-model-pv-d968e7b0 in namespace e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig router-managed-llmisvc-model-pv-d968e7b0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig router-managed-llmisvc-model-pv-d968e7b0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig workload-pd-cpu-llmisvc-model-p-9d807ba3 in namespace e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig workload-pd-cpu-llmisvc-model-p-9d807ba3 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig workload-pd-cpu-llmisvc-model-p-9d807ba3 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig model-pvc-llmisvc-model-pvc-rou-49c1f027 in namespace e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig model-pvc-llmisvc-model-pvc-rou-49c1f027 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig model-pvc-llmisvc-model-pvc-rou-49c1f027 [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_inference_service] [2026-07-28T10:55:06.102444] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-pd-cpu', 'model-pvc'], prompt='KServe is a', service_name='llmisvc-model-pvc-router-manage-e8706282', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, namespace='e2e-test-llm-inference-service-48639af5', before_test=[ at 0x7f7bf97ff380>], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-pvc-router-manage-e8706282', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-inference-service-48639af5', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-pv-d968e7b0'}, [e2e-llm-inference-service] {'name': 'workload-pd-cpu-llmisvc-model-p-9d807ba3'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-49c1f027'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-28T10:55:06.114770] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-pvc-router-manage-e8706282', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-inference-service-48639af5', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-pv-d968e7b0'}, [e2e-llm-inference-service] {'name': 'workload-pd-cpu-llmisvc-model-p-9d807ba3'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-49c1f027'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-28T10:55:06.253139] end - ✅ in 0.138s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-28T10:55:06.253418] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-pvc-router-manage-e8706282', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-inference-service-48639af5', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-pv-d968e7b0'}, [e2e-llm-inference-service] {'name': 'workload-pd-cpu-llmisvc-model-p-9d807ba3'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-49c1f027'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'RouterReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T10:55:21Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T10:55:21Z', 'message': 'Inference Pool e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T10:55:21Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T10:55:21Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-28T10:55:21Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T10:55:21Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T10:55:21Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T10:55:21Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T10:55:21Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'RouterReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T10:55:30Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T10:55:30Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T10:55:30Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T10:55:30Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-28T10:55:21Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T10:55:30Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T10:55:30Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T10:55:21Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T10:55:30Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T10:55:30Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T10:55:30Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T10:55:30Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T10:55:30Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-28T10:55:21Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T10:55:30Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T10:55:52Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T10:55:52Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T10:55:30Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T10:55:30Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T10:55:30Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T10:57:33Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T10:55:30Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-28T10:55:21Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T10:55:30Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T10:55:52Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T10:55:52Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T10:55:30Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1391 Timed out waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T10:55:30Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T10:55:30Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T10:57:33Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T10:55:30Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-28T10:55:21Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T10:55:30Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T10:55:52Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T10:55:52Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T10:55:30Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-07-28T11:10:06.635359] end - ❌ 900.381s: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T10:55:30Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T10:55:30Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T10:57:33Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T10:55:30Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-28T10:55:21Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T10:55:30Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T10:55:52Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T10:55:52Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T10:55:30Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:903 [router-managed-workload-pd-cpu-model-pvc] ❌ ERROR: Failed to call llm inference service llmisvc-model-pvc-router-manage-e8706282: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T10:55:30Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T10:55:30Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T10:57:33Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T10:55:30Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-28T10:55:21Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T10:55:30Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T10:55:52Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T10:55:52Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T10:55:30Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:243 # Diagnostics for 'llmisvc-model-pvc-router-manage-e8706282' in 'e2e-test-llm-inference-service-48639af5' [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:244 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:247 # LLMInferenceService llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:256 apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] security.opendatahub.io/enable-auth: 'false' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T10:55:06Z' [e2e-llm-inference-service] finalizers: [e2e-llm-inference-service] - serving.kserve.io/llmisvc-finalizer [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:security.opendatahub.io/enable-auth: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:baseRefs: {} [e2e-llm-inference-service] manager: OpenAPI-Generator [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T10:55:06Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:finalizers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] v:"serving.kserve.io/llmisvc-finalizer": {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T10:55:06Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:addresses: {} [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-router-route: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-scheduler: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-scheduler-latency-predictor: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-tokenizer: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-tracing: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-worker-data-parallel: {} [e2e-llm-inference-service] f:appliedConfigs: {} [e2e-llm-inference-service] f:conditions: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:router: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:gateways: {} [e2e-llm-inference-service] f:scheduler: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:inferencePool: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:service: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:url: {} [e2e-llm-inference-service] f:workloads: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:prefill: {} [e2e-llm-inference-service] f:primary: {} [e2e-llm-inference-service] f:scheduler: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-28T10:57:33Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] resourceVersion: '59665' [e2e-llm-inference-service] uid: a31a8e8c-0db4-4e00-98bc-e1edb1da6f1e [e2e-llm-inference-service] spec: [e2e-llm-inference-service] baseRefs: [e2e-llm-inference-service] - name: router-managed-llmisvc-model-pv-d968e7b0 [e2e-llm-inference-service] - name: workload-pd-cpu-llmisvc-model-p-9d807ba3 [e2e-llm-inference-service] - name: model-pvc-llmisvc-model-pvc-rou-49c1f027 [e2e-llm-inference-service] model: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uri: '' [e2e-llm-inference-service] status: [e2e-llm-inference-service] addresses: [e2e-llm-inference-service] - name: gateway-external-model-routing [e2e-llm-inference-service] url: http://ac8a2e4f596c64362acf1a34ef0ad9a3-2038940287.us-east-1.elb.amazonaws.com/ [e2e-llm-inference-service] - name: gateway-external [e2e-llm-inference-service] url: http://ac8a2e4f596c64362acf1a34ef0ad9a3-2038940287.us-east-1.elb.amazonaws.com/e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] - name: gateway-external [e2e-llm-inference-service] url: http://ac8a2e4f596c64362acf1a34ef0ad9a3-2038940287.us-east-1.elb.amazonaws.com/publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] - name: gateway-internal-model-routing [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/ [e2e-llm-inference-service] - name: gateway-internal [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] - name: gateway-internal [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-template: kserve-config-llm-decode-template [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-worker-data-parallel: kserve-config-llm-decode-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-template: kserve-config-llm-prefill-template [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-worker-data-parallel: kserve-config-llm-prefill-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-router-route: kserve-config-llm-router-route [e2e-llm-inference-service] serving.kserve.io/config-llm-scheduler: kserve-config-llm-scheduler [e2e-llm-inference-service] serving.kserve.io/config-llm-scheduler-latency-predictor: kserve-config-llm-scheduler-latency-predictor [e2e-llm-inference-service] serving.kserve.io/config-llm-template: kserve-config-llm-template [e2e-llm-inference-service] serving.kserve.io/config-llm-tokenizer: kserve-config-llm-tokenizer [e2e-llm-inference-service] serving.kserve.io/config-llm-tracing: kserve-config-llm-tracing [e2e-llm-inference-service] serving.kserve.io/config-llm-worker-data-parallel: kserve-config-llm-worker-data-parallel [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T10:55:30Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: HTTPRoutesReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T10:55:30Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: InferencePoolReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T10:57:33Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: MainWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T10:55:30Z' [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: PrefillWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T10:55:21Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: PresetsCombined [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T10:55:30Z' [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: Ready [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T10:55:52Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: RouterReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T10:55:52Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: SchedulerWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T10:55:30Z' [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: WorkloadsReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] url: http://ac8a2e4f596c64362acf1a34ef0ad9a3-2038940287.us-east-1.elb.amazonaws.com/e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:68 TIME NAMESPACE SOURCE TYPE REASON MESSAGE [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:69 -------------------------------------------------------------------------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None e2e-test-llm-inference-service-48639af5 Normal Scheduled Successfully assigned e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-2sw7r to ip-10-0-131-88.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 10:54:51 e2e-test-llm-inference-service-48639af5 attachdetach-controller Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-f219be45-903a-4603-991c-a0a969f33047" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 10:54:54 e2e-test-llm-inference-service-48639af5 multus Normal AddedInterface Add eth0 [10.133.0.61/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 10:54:54 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-131-88.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:527f9c4ef06f4e1d255c6c4b87333be01c67680fe0b3417b8fb7aeb5d5b61bbf" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 10:54:54 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-131-88.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 10:54:54 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-131-88.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 10:54:45 e2e-test-llm-inference-service-48639af5 job-controller Normal SuccessfulCreate Created pod: e2e-pvc-model-download-2sw7r [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 10:55:03 e2e-test-llm-inference-service-48639af5 job-controller Normal Completed Job completed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 10:54:45 e2e-test-llm-inference-service-48639af5 persistentvolume-controller Normal WaitForFirstConsumer waiting for first consumer to be created before binding [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 10:54:45 e2e-test-llm-inference-service-48639af5 ebs.csi.aws.com_aws-ebs-csi-driver-controller-d96bdd68d-rh6ng_afb65174-87af-46ee-b050-57713714920e Normal Provisioning External provisioner is provisioning volume for claim "e2e-test-llm-inference-service-48639af5/e2e-pvc-model-storage" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 10:54:47 e2e-test-llm-inference-service-48639af5 persistentvolume-controller Normal ExternalProvisioning Waiting for a volume to be created either by the external provisioner 'ebs.csi.aws.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 10:54:48 e2e-test-llm-inference-service-48639af5 ebs.csi.aws.com_aws-ebs-csi-driver-controller-d96bdd68d-rh6ng_afb65174-87af-46ee-b050-57713714920e Normal ProvisioningSucceeded Successfully provisioned volume pvc-f219be45-903a-4603-991c-a0a969f33047 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 10:55:19 e2e-test-llm-inference-service-48639af5 replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None e2e-test-llm-inference-service-48639af5 Normal Scheduled Successfully assigned e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv to ip-10-0-131-88.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 10:55:21 e2e-test-llm-inference-service-48639af5 attachdetach-controller Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-f219be45-903a-4603-991c-a0a969f33047" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 10:55:24 e2e-test-llm-inference-service-48639af5 multus Normal AddedInterface Add eth0 [10.133.0.62/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 10:55:24 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-131-88.ec2.internal Normal Pulled Container image "quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.9.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 10:55:24 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-131-88.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 10:55:24 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-131-88.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 10:55:24 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-131-88.ec2.internal Normal Pulled Container image "vllm/vllm-openai-cpu:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 10:55:24 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-131-88.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 10:55:24 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-131-88.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 10:57:14 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-131-88.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.133.0.62:8001/health": dial tcp 10.133.0.62:8001: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 10:55:19 e2e-test-llm-inference-service-48639af5 replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None e2e-test-llm-inference-service-48639af5 Normal Scheduled Successfully assigned e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln to ip-10-0-141-50.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 10:55:21 e2e-test-llm-inference-service-48639af5 attachdetach-controller Warning FailedAttachVolume Multi-Attach error for volume "pvc-f219be45-903a-4603-991c-a0a969f33047" Volume is already used by pod(s) llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 10:55:19 e2e-test-llm-inference-service-48639af5 deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b89cd7957 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 10:55:19 e2e-test-llm-inference-service-48639af5 deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 10:55:16 e2e-test-llm-inference-service-48639af5 OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-pvc-router-manage-e8706282-kserve-route-authn: failed to get HTTPRoute e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-pvc-router-manage-e8706282-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 10:55:18 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.Secret e2e-test-llm-inference-service-48639af5/llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 10:55:18 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.ServiceAccount e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 10:55:18 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.Role e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 10:55:19 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.RoleBinding e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 10:55:19 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.Deployment e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 10:55:19 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.Deployment e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 10:55:19 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.Service e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 10:55:19 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.ServiceAccount e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 10:55:19 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /e2e-test-llm-inference-service-132820f03254470cd8a0146008974a5f [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 10:55:29 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 11:05:40 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Updated Updated v1.Secret e2e-test-llm-inference-service-48639af5/llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 10:55:29 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None e2e-test-llm-inference-service-48639af5 Normal Scheduled Successfully assigned e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche5mbz2 to ip-10-0-131-88.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 10:55:19 e2e-test-llm-inference-service-48639af5 multus Normal AddedInterface Add eth0 [10.133.0.63/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 10:55:19 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-131-88.ec2.internal Normal Pulled Container image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 10:55:19 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-131-88.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 10:55:19 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-131-88.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 10:55:19 e2e-test-llm-inference-service-48639af5 replicaset-controller Normal SuccessfulCreate Created pod: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche5mbz2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 10:55:19 e2e-test-llm-inference-service-48639af5 deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-7fc69df7b8 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:172 ### Pod llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:212 #### init-container 'llm-d-routing-sidecar' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:225 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:226 {"level":"info","ts":1785236124.4676976,"msg":"Proxy starting","Built on":"","From Git SHA":"unknown"} [e2e-llm-inference-service] {"level":"info","ts":1785236124.4677277,"msg":"Proxy configuration","config":"{\"Port\":\"8000\",\"KVConnector\":\"nixlv2\",\"ECConnector\":\"\",\"DataParallelSize\":1,\"MaxIdleConnsPerHost\":1024,\"EnablePrefillerSampling\":false,\"PrefillMaxRetries\":0,\"PrefillRetryBackoff\":200000000,\"UseTLSForPrefiller\":true,\"UseTLSForDecoder\":true,\"UseTLSForEncoder\":false,\"InsecureSkipVerifyForPrefiller\":false,\"InsecureSkipVerifyForEncoder\":false,\"InsecureSkipVerifyForDecoder\":false,\"SecureServing\":true,\"CertPath\":\"/var/run/kserve/tls\",\"MooncakeBootstrapPort\":8998,\"EnableSSRFProtection\":true,\"InferencePoolNamespace\":\"e2e-test-llm-inference-service-48639af5\",\"InferencePoolName\":\"llmisvc-model-pvc-router-manage-e8706282-inference-pool\",\"PoolGroup\":\"inference.networking.x-k8s.io\",\"DecodeChunkSize\":0,\"Tracing\":false,\"MoRIIOWriteMode\":false,\"MoRIIODecodeNotifyPort\":61005,\"MoRIIODecodeHandshakePort\":6301,\"MoRIIODecodePodIP\":\"\",\"MoRIIOParallelDispatch\":false,\"MoRIIOPrefillHandshakePort\":6301,\"MoRIIOPrefillNotifyPort\":61005,\"MoRIIOTPSize\":1,\"MoRIIODPSize\":1,\"MoRIIORemoteHosts\":null,\"MoRIIODPSizeLocal\":0,\"MoRIIODecodeHosts\":null,\"DecoderURL\":\"https://localhost:8001\"}"} [e2e-llm-inference-service] {"level":"info","ts":1785236124.4683309,"logger":"allowlist-validator","msg":"starting SSRF protection allowlist validator","namespace":"e2e-test-llm-inference-service-48639af5","poolName":"llmisvc-model-pvc-router-manage-e8706282-inference-pool","gvr":"inference.networking.x-k8s.io/v1alpha2, Resource=inferencepools"} [e2e-llm-inference-service] {"level":"info","ts":1785236124.4831393,"logger":"allowlist-validator","msg":"InferencePool added","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785236124.4923198,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":0,"targets":{}} [e2e-llm-inference-service] {"level":"info","ts":1785236124.4923873,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":0,"targets":{}} [e2e-llm-inference-service] {"level":"info","ts":1785236124.569008,"logger":"allowlist-validator","msg":"allowlist validator started successfully"} [e2e-llm-inference-service] {"level":"info","ts":1785236124.570147,"logger":"proxy server on port 8000","msg":"server TLS configured"} [e2e-llm-inference-service] {"level":"info","ts":1785236124.5701632,"logger":"proxy server on port 8000","msg":"starting","addr":"[::]:8000"} [e2e-llm-inference-service] {"level":"info","ts":1785236124.6130328,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.62":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785236125.62399,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.62":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785236134.2901926,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.62":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785236154.48376,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785236154.4973652,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.62":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785236154.4974246,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.62":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785236184.4843094,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785236184.4939806,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.62":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785236184.494045,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.62":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785236214.484902,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785236214.4971745,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.62":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785236214.4972355,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.62":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785236244.2903726,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.62":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785236244.3073962,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.62":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785236244.485515,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785236244.4934547,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.62":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785236244.49352,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.62":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785236274.4862425,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785236274.496628,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.62":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785236274.4967241,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.62":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785236304.4871802,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785236304.4971747,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.62":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785236304.4972372,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.62":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785236334.4876244,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785236334.4971282,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.62":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785236334.4971805,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.62":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785236364.4878664,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785236364.4980414,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.62":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785236364.4980931,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.62":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785236394.4887238,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785236394.4986243,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.62":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785236394.498678,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.62":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785236424.4890292,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785236424.4994516,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.62":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785236424.4995153,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.62":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785236454.4897683,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785236454.499692,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.62":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785236454.4997604,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.62":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785236484.4900215,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785236484.5001495,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.62":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785236484.5002077,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.62":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785236514.4902623,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785236514.4999893,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.62":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785236514.5000415,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.62":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785236544.4906924,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785236544.5003822,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.62":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785236544.5004432,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.62":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785236574.491652,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785236574.5018263,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.62":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785236574.5018892,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.62":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785236604.4918723,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785236604.500483,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.62":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785236604.500542,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.62":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785236634.4923637,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785236634.501956,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.62":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785236634.5020137,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.62":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785236664.4930034,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785236664.501566,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.62":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785236664.5016327,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.62":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785236694.4937499,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785236694.5043473,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.62":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785236694.5044074,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.62":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785236724.4947078,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785236724.5058014,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.62":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785236724.505861,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.62":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785236754.495718,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785236754.5055873,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.62":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785236754.5056615,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.62":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785236784.496335,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785236784.5064116,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.62":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785236784.5064697,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.62":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785236814.4969473,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785236814.5074315,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.62":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785236814.5074797,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.62":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785236844.49751,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785236844.5078778,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.62":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785236844.5079532,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.62":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785236874.4976737,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785236874.5080655,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.62":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785236874.5081263,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.62":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785236904.4977782,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785236904.5072963,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.62":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785236904.507361,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.62":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785236934.498571,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785236934.5100572,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.62":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785236934.5101097,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.62":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785236964.4989238,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785236964.5080798,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.62":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785236964.5081432,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.62":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785236994.4992533,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785236994.5098023,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.62":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785236994.5098832,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.62":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv":{}}} [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:212 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:225 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:226 (EngineCore pid=74) INFO 07-28 10:56:37 [v1/worker/cpu_worker.py:236] auto thread-binding list (id, physical core): [(4, 0), (5, 1), (6, 2), (7, 3)] [e2e-llm-inference-service] [W728 10:56:37.140084179 utils.cpp:76] Warning: numa_migrate_pages failed. errno: 1 (function init_cpu_threads_env) [e2e-llm-inference-service] [W728 10:56:37.140108908 utils.cpp:103] Warning: NUMA binding: Using MEMBIND policy for memory allocation on the NUMA nodes (0). Memory allocations will be strictly bound to these NUMA nodes. (function init_cpu_threads_env) [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-28 10:56:37 [v1/worker/cpu_worker.py:109] OMP threads binding of Process 74: [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-28 10:56:37 [v1/worker/cpu_worker.py:109] OMP tid: 74, core 4 [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-28 10:56:37 [v1/worker/cpu_worker.py:109] OMP tid: 91, core 5 [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-28 10:56:37 [v1/worker/cpu_worker.py:109] OMP tid: 92, core 6 [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-28 10:56:37 [v1/worker/cpu_worker.py:109] OMP tid: 93, core 7 [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-28 10:56:37 [v1/worker/cpu_worker.py:109] [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-28 10:56:37 [distributed/parallel_state.py:1356] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.133.0.62:53915 backend=gloo [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-28 10:56:37 [distributed/parallel_state.py:1400] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.133.0.62:53915 backend=gloo [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-28 10:56:37 [distributed/parallel_state.py:1459] Detected 1 nodes in the distributed environment [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-28 10:56:37 [distributed/parallel_state.py:1716] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-28 10:56:37 [v1/sample/logits_processor/__init__.py:65] No logitsprocs plugins installed (group vllm.logits_processors). [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-28 10:56:37 [model_executor/offloader/base.py:107] Offloader set to NoopOffloader (no offloading). [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-28 10:56:37 [v1/worker/cpu_model_runner.py:71] Starting to load model /mnt/models... [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-28 10:56:37 [compilation/decorators.py:213] Inferred dynamic dimensions for forward method of : ['input_ids', 'positions', 'intermediate_tensors', 'inputs_embeds'] [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-28 10:56:37 [config/compilation.py:1194] enabled custom ops: Counter() [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-28 10:56:37 [config/compilation.py:1195] disabled custom ops: Counter({'vocab_parallel_embedding': 1, 'logits_processor': 1}) [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-28 10:56:37 [model_executor/model_loader/base_loader.py:63] Loading weights on cpu ... [e2e-llm-inference-service] (EngineCore pid=74) Loading pt checkpoint shards: 0% Completed | 0/1 [00:00 [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 10:56:46 [v1/engine/utils.py:1047] Waiting for 1 local, 0 remote core engine proc(s) to start. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 10:56:56 [v1/engine/utils.py:1047] Waiting for 1 local, 0 remote core engine proc(s) to start. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 10:57:06 [v1/engine/utils.py:1047] Waiting for 1 local, 0 remote core engine proc(s) to start. [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-28 10:57:16 [compilation/decorators.py:640] saved AOT compiled function to /home/.cache/vllm/torch_compile_cache/torch_aot_compile/86c9c3c579382eef68a98ac1d59b39811ba08abef3b4e90675a32c8dec3d7c90/rank_0_0/model [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 10:57:16 [v1/engine/utils.py:1047] Waiting for 1 local, 0 remote core engine proc(s) to start. [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-28 10:57:17 [compilation/monitor.py:76] Initial profiling/warmup run took 1.01 s [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-28 10:57:17 [v1/worker/cpu_model_runner.py:92] Warming up done. [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-28 10:57:17 [v1/engine/core.py:283] init engine (profile, create kv cache, warmup model) took 38.95 seconds [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-28 10:57:17 [tokenizers/registry.py:68] Loading CachedHfTokenizer for tokenizer_mode='hf' [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-28 10:57:17 [utils/gc_utils.py:40] GC Debug Config. enabled:False,top_objects:-1 [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-28 10:57:17 [config/vllm.py:790] Asynchronous scheduling is disabled. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 10:57:17 [v1/engine/utils.py:1158] READY from local core engine process 0. [e2e-llm-inference-service] (EngineCore pid=74) WARNING 07-28 10:57:17 [config/vllm.py:859] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-28 10:57:17 [v1/engine/core.py:1158] EngineCore waiting for work. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 10:57:17 [v1/metrics/loggers.py:273] Engine 000: vllm cache_config_info with initialization after num_gpu_blocks is: 227 [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-28 10:57:17 [v1/engine/core.py:1158] EngineCore waiting for work. [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-28 10:57:17 [v1/engine/core.py:1158] EngineCore waiting for work. [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 10:57:17 [entrypoints/openai/api_server.py:590] Supported tasks: ['generate'] [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 10:57:18 [renderers/base.py:197] Warming up chat template processing... [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 10:57:18 [renderers/hf.py:122] Failed to load AutoTokenizer chat template for /mnt/models [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 10:57:18 [renderers/hf.py:122] Traceback (most recent call last): [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 10:57:18 [renderers/hf.py:122] File "/opt/venv/lib/python3.12/site-packages/vllm/renderers/hf.py", line 120, in resolve_chat_template [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 10:57:18 [renderers/hf.py:122] return tokenizer.get_chat_template(chat_template, tools=tools) [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 10:57:18 [renderers/hf.py:122] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 10:57:18 [renderers/hf.py:122] File "/opt/venv/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 1825, in get_chat_template [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 10:57:18 [renderers/hf.py:122] raise ValueError( [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 10:57:18 [renderers/hf.py:122] ValueError: Cannot use chat template functions because tokenizer.chat_template is not set and no template argument was passed! For information about writing templates and setting the tokenizer.chat_template attribute, please see the documentation at https://huggingface.co/docs/transformers/main/en/chat_templating [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 10:57:18 [renderers/hf.py:141] There is no chat template fallback for /mnt/models [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 10:57:18 [renderers/hf.py:314] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 10:57:18 [renderers/hf.py:122] Failed to load AutoTokenizer chat template for /mnt/models [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 10:57:18 [renderers/hf.py:122] Traceback (most recent call last): [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 10:57:18 [renderers/hf.py:122] File "/opt/venv/lib/python3.12/site-packages/vllm/renderers/hf.py", line 120, in resolve_chat_template [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 10:57:18 [renderers/hf.py:122] return tokenizer.get_chat_template(chat_template, tools=tools) [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 10:57:18 [renderers/hf.py:122] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 10:57:18 [renderers/hf.py:122] File "/opt/venv/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 1825, in get_chat_template [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 10:57:18 [renderers/hf.py:122] raise ValueError( [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 10:57:18 [renderers/hf.py:122] ValueError: Cannot use chat template functions because tokenizer.chat_template is not set and no template argument was passed! For information about writing templates and setting the tokenizer.chat_template attribute, please see the documentation at https://huggingface.co/docs/transformers/main/en/chat_templating [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 10:57:18 [renderers/base.py:205] This model does not support chat template. [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 10:57:18 [entrypoints/openai/api_server.py:594] Starting vLLM server on https://0.0.0.0:8001 [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 10:57:18 [entrypoints/launcher.py:37] Available routes are: [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 10:57:18 [entrypoints/launcher.py:46] Route: /openapi.json, Methods: GET, HEAD [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 10:57:18 [entrypoints/launcher.py:46] Route: /docs, Methods: GET, HEAD [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 10:57:18 [entrypoints/launcher.py:46] Route: /docs/oauth2-redirect, Methods: GET, HEAD [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 10:57:18 [entrypoints/launcher.py:46] Route: /redoc, Methods: GET, HEAD [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 10:57:18 [entrypoints/launcher.py:46] Route: /tokenize, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 10:57:18 [entrypoints/launcher.py:46] Route: /detokenize, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 10:57:18 [entrypoints/launcher.py:46] Route: /load, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 10:57:18 [entrypoints/launcher.py:46] Route: /version, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 10:57:18 [entrypoints/launcher.py:46] Route: /health, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 10:57:18 [entrypoints/launcher.py:46] Route: /metrics, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 10:57:18 [entrypoints/launcher.py:46] Route: /v1/models, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 10:57:18 [entrypoints/launcher.py:46] Route: /ping, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 10:57:18 [entrypoints/launcher.py:46] Route: /ping, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 10:57:18 [entrypoints/launcher.py:46] Route: /invocations, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 10:57:18 [entrypoints/launcher.py:46] Route: /v1/chat/completions, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 10:57:18 [entrypoints/launcher.py:46] Route: /v1/chat/completions/batch, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 10:57:18 [entrypoints/launcher.py:46] Route: /v1/responses, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 10:57:18 [entrypoints/launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 10:57:18 [entrypoints/launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 10:57:18 [entrypoints/launcher.py:46] Route: /v1/completions, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 10:57:18 [entrypoints/launcher.py:46] Route: /v1/messages, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 10:57:18 [entrypoints/launcher.py:46] Route: /v1/messages/count_tokens, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 10:57:18 [entrypoints/launcher.py:46] Route: /inference/v1/generate, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 10:57:18 [entrypoints/launcher.py:46] Route: /scale_elastic_ep, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 10:57:18 [entrypoints/launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 10:57:18 [entrypoints/launcher.py:46] Route: /v1/chat/completions/render, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 10:57:18 [entrypoints/launcher.py:46] Route: /v1/completions/render, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO: Started server process [1] [e2e-llm-inference-service] (APIServer pid=1) INFO: Waiting for application startup. [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 10:57:18 [entrypoints/ssl.py:60] SSLCertRefresher monitors files: ['/var/run/kserve/tls/tls.key', '/var/run/kserve/tls/tls.crt'] [e2e-llm-inference-service] (APIServer pid=1) INFO: Application startup complete. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 10:57:24 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 10:57:24 [entrypoints/ssl.py:64] File change detected: modified - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 10:57:24 [entrypoints/ssl.py:34] Reloading SSL certificate chain [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 10:57:24 [entrypoints/ssl.py:64] File change detected: modified - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 10:57:24 [entrypoints/ssl.py:34] Reloading SSL certificate chain [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 10:57:24 [entrypoints/ssl.py:64] File change detected: deleted - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 10:57:24 [entrypoints/ssl.py:34] Reloading SSL certificate chain [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 10:57:24 [entrypoints/ssl.py:64] File change detected: deleted - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 10:57:24 [entrypoints/ssl.py:34] Reloading SSL certificate chain [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 10:57:28 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 10:57:38 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 10:57:48 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 10:57:58 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 10:58:08 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 10:58:18 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 10:58:28 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 10:58:38 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 10:58:48 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 10:58:58 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 10:59:08 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 10:59:18 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 10:59:28 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 10:59:38 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 10:59:48 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 10:59:58 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 11:00:08 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 11:00:18 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 11:00:28 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 11:00:38 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 11:00:48 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 11:00:58 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 11:01:08 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 11:01:18 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 11:01:28 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 11:01:38 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 11:01:48 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 11:01:58 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 11:02:08 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 11:02:18 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 11:02:28 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 11:02:38 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 11:02:48 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 11:02:58 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 11:03:08 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 11:03:18 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 11:03:28 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 11:03:38 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 11:03:48 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 11:03:58 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 11:04:08 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 11:04:18 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 11:04:28 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 11:04:38 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 11:04:48 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 11:04:58 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 11:05:08 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 11:05:18 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 11:05:28 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 11:05:38 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 11:05:48 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 11:05:58 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 11:06:08 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 11:06:18 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 11:06:28 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 11:06:38 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 11:06:48 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 11:06:58 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 11:07:08 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 11:07:18 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 11:07:28 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 11:07:38 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 11:07:48 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 11:07:58 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 11:08:08 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 11:08:18 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 11:08:28 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 11:08:38 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 11:08:48 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 11:08:58 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 11:09:08 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 11:09:18 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 11:09:28 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 11:09:38 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 11:09:48 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 11:09:58 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:172 ### Pod llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln (phase=Pending) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:212 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:228 # -- logs (current): unavailable ((400) [e2e-llm-inference-service] Reason: Bad Request [e2e-llm-inference-service] HTTP response headers: HTTPHeaderDict({'Audit-Id': 'f831f3bd-a3b3-4d45-be12-ab480f51dfaf', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'Date': 'Tue, 28 Jul 2026 11:10:06 GMT', 'Content-Length': '247'}) [e2e-llm-inference-service] HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"container \"main\" in pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln\" is waiting to start: ContainerCreating","reason":"BadRequest","code":400} [e2e-llm-inference-service] [e2e-llm-inference-service] ) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:172 ### Pod llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche5mbz2 (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:212 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:225 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:226 {"level":"info","ts":1785236119.9605606,"logger":"setup","caller":"runner/runner.go:196","msg":"GIE build","commit-sha":"unknown","build-ref":""} [e2e-llm-inference-service] {"level":"info","ts":1785236119.9607735,"logger":"setup","caller":"runner/runner.go:217","msg":"Flags processed","flags":{"cert-path":"/var/run/kserve/tls","config-file":"","config-text":"apiVersion: llm-d.ai/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n- type: disagg-headers-handler\n- type: prefill-filter\n- type: decode-filter\n- type: queue-scorer\n- type: kv-cache-utilization-scorer\n- type: active-request-scorer\n- type: prefix-cache-scorer\n- type: max-score-picker\n- type: always-disagg-pd-decider\n- parameters:\n deciders:\n prefill: always-disagg-pd-decider\n type: disagg-profile-handler\n- parameters:\n scheme: https\n type: metrics-data-source\nschedulingProfiles:\n- name: prefill\n plugins:\n - pluginRef: prefill-filter\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: queue-scorer\n weight: 2\n - pluginRef: kv-cache-utilization-scorer\n weight: 2\n - pluginRef: max-score-picker\n- name: decode\n plugins:\n - pluginRef: decode-filter\n - pluginRef: active-request-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n","disable-endpoint-subset-filter":false,"enable-cert-reload":true,"enable-grpc-stream-metrics":false,"enable-pprof":true,"endpoint-selector":"","endpoint-target-ports":{},"grpc-health-port":9003,"grpc-max-recv-msg-size":"","grpc-max-send-msg-size":"","grpc-port":9002,"ha-enable-leader-election":false,"health-checking":false,"metrics-endpoint-auth":true,"metrics-port":9090,"metrics-staleness-threshold":2000000000,"model-server-metrics-https-insecure-skip-verify":true,"model-server-metrics-path":"/metrics","model-server-metrics-port":0,"model-server-metrics-scheme":"http","pool-group":"inference.networking.k8s.io","pool-name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool","pool-namespace":"e2e-test-llm-inference-service-48639af5","refresh-metrics-interval":50000000,"refresh-prometheus-metrics-interval":5000000000,"secure-serving":true,"tracing":true,"v":2,"zap-devel":{},"zap-encoder":{},"zap-log-level":{},"zap-stacktrace-level":{},"zap-time-encoding":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785236119.960969,"logger":"setup.trace","caller":"tracing/telemetry.go:123","msg":"init OTel trace exporter","type":"console"} [e2e-llm-inference-service] {"level":"info","ts":1785236119.9617152,"caller":"loader/configloader.go:121","msg":"Loaded raw configuration","config":"{Plugins: [{Type: disagg-headers-handler} {Type: prefill-filter} {Type: decode-filter} {Type: queue-scorer} {Type: kv-cache-utilization-scorer} {Type: active-request-scorer} {Type: prefix-cache-scorer} {Type: max-score-picker} {Type: always-disagg-pd-decider} {Type: disagg-profile-handler, Parameters: {\"deciders\":{\"prefill\":\"always-disagg-pd-decider\"}}} {Type: metrics-data-source, Parameters: {\"scheme\":\"https\"}}], SchedulingProfiles: [{Name: prefill, Plugins: [{PluginRef: prefill-filter} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: queue-scorer, Weight: 2.00} {PluginRef: kv-cache-utilization-scorer, Weight: 2.00} {PluginRef: max-score-picker}]} {Name: decode, Plugins: [{PluginRef: decode-filter} {PluginRef: active-request-scorer, Weight: 2.00} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: max-score-picker}]}]}"} [e2e-llm-inference-service] {"level":"info","ts":1785236119.9617503,"logger":"setup","caller":"runner/runner.go:622","msg":"Data layer: ENABLED"} [e2e-llm-inference-service] {"level":"info","ts":1785236119.9620233,"logger":"setup","caller":"runner/runner.go:281","msg":"Raw config after phase one","config":{"apiVersion":"llm-d.ai/v1alpha1","dataLayer":null,"kind":"EndpointPickerConfig","plugins":[{"name":"disagg-headers-handler","parameters":null,"type":"disagg-headers-handler"},{"name":"prefill-filter","parameters":null,"type":"prefill-filter"},{"name":"decode-filter","parameters":null,"type":"decode-filter"},{"name":"queue-scorer","parameters":null,"type":"queue-scorer"},{"name":"kv-cache-utilization-scorer","parameters":null,"type":"kv-cache-utilization-scorer"},{"name":"active-request-scorer","parameters":null,"type":"active-request-scorer"},{"name":"prefix-cache-scorer","parameters":null,"type":"prefix-cache-scorer"},{"name":"max-score-picker","parameters":null,"type":"max-score-picker"},{"name":"always-disagg-pd-decider","parameters":null,"type":"always-disagg-pd-decider"},{"name":"disagg-profile-handler","parameters":{"deciders":{"prefill":"always-disagg-pd-decider"}},"type":"disagg-profile-handler"},{"name":"metrics-data-source","parameters":{"scheme":"https"},"type":"metrics-data-source"}],"schedulingProfiles":[{"name":"prefill","plugins":[{"pluginRef":"prefill-filter","weight":null},{"pluginRef":"prefix-cache-scorer","weight":3},{"pluginRef":"queue-scorer","weight":2},{"pluginRef":"kv-cache-utilization-scorer","weight":2},{"pluginRef":"max-score-picker","weight":null}]},{"name":"decode","plugins":[{"pluginRef":"decode-filter","weight":null},{"pluginRef":"active-request-scorer","weight":2},{"pluginRef":"prefix-cache-scorer","weight":3},{"pluginRef":"max-score-picker","weight":null}]}]}} [e2e-llm-inference-service] {"level":"info","ts":1785236119.9810796,"caller":"disagg/disagg_profile_handler.go:186","msg":"No deciders.encode configured, E disaggregation disabled"} [e2e-llm-inference-service] {"level":"info","ts":1785236119.9812074,"logger":"utilization-detector/utilization-detector","caller":"utilization/detector.go:83","msg":"Creating new UtilizationDetector","queueDepthThreshold":5,"kvCacheUtilThreshold":0.8,"metricsStalenessThreshold":"200ms","headroom":0} [e2e-llm-inference-service] {"level":"info","ts":1785236119.9813135,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"vllm","mapping":"Mapping{all specs enabled}"} [e2e-llm-inference-service] {"level":"info","ts":1785236119.9813676,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"sglang","mapping":"Mapping{disabled: [lora]}"} [e2e-llm-inference-service] {"level":"info","ts":1785236119.9814174,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"trtllm-serve","mapping":"Mapping{disabled: [lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1785236119.9814909,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"triton-tensorrt-llm","mapping":"Mapping{disabled: [lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1785236119.981524,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"triton","mapping":"Mapping{disabled: [kv, lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1785236119.9818459,"caller":"loader/configloader.go:154","msg":"Instantiated all plugins and applied system defaults. Effective raw configuration","config":"{Plugins: [{Name: disagg-headers-handler, Type: disagg-headers-handler} {Name: prefill-filter, Type: prefill-filter} {Name: decode-filter, Type: decode-filter} {Name: queue-scorer, Type: queue-scorer} {Name: kv-cache-utilization-scorer, Type: kv-cache-utilization-scorer} {Name: active-request-scorer, Type: active-request-scorer} {Name: prefix-cache-scorer, Type: prefix-cache-scorer} {Name: max-score-picker, Type: max-score-picker} {Name: always-disagg-pd-decider, Type: always-disagg-pd-decider} {Name: disagg-profile-handler, Type: disagg-profile-handler, Parameters: {\"deciders\":{\"prefill\":\"always-disagg-pd-decider\"}}} {Name: metrics-data-source, Type: metrics-data-source, Parameters: {\"scheme\":\"https\"}} {Name: fcfs-ordering-policy, Type: fcfs-ordering-policy} {Name: global-strict-fairness-policy, Type: global-strict-fairness-policy} {Name: static-usage-limit-policy, Type: static-usage-limit-policy} {Name: openai-parser, Type: openai-parser} {Name: anthropic-parser, Type: anthropic-parser} {Name: vllmhttp-parser, Type: vllmhttp-parser} {Name: utilization-detector, Type: utilization-detector} {Name: core-metrics-extractor, Type: core-metrics-extractor}], SchedulingProfiles: [{Name: prefill, Plugins: [{PluginRef: prefill-filter} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: queue-scorer, Weight: 2.00} {PluginRef: kv-cache-utilization-scorer, Weight: 2.00} {PluginRef: max-score-picker}]} {Name: decode, Plugins: [{PluginRef: decode-filter} {PluginRef: active-request-scorer, Weight: 2.00} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: max-score-picker}]}], DataLayer: {Sources: [{PluginRef: metrics-data-source, Extractors: [{PluginRef: core-metrics-extractor}]}], Discovery: }, FlowControl: {MaxBytes: unlimited, MaxRequests: unlimited, SaturationDetector: {PluginRef: utilization-detector}}, RequestHandler: {Parsers: [{PluginRef: openai-parser}, {PluginRef: anthropic-parser}, {PluginRef: vllmhttp-parser}]}}"} [e2e-llm-inference-service] {"level":"info","ts":1785236119.983452,"caller":"approximateprefix/plugin.go:88","msg":"Prefix DataProducer initialized","config":{"autoTune":true,"blockSizeTokens":16,"blockSize":0,"maxPrefixBlocksToMatch":2048,"maxPrefixTokensToMatch":131072,"lruCapacityPerServer":31250}} [e2e-llm-inference-service] {"level":"info","ts":1785236119.9835386,"caller":"approximateprefix/plugin.go:111","msg":"WARNING: configured blockSizeTokens is below the recommended minimum, overriding it.","blockSizeTokens":16,"minimum":64,"issue":"https://github.com/llm-d/llm-d-router/issues/1158"} [e2e-llm-inference-service] {"level":"info","ts":1785236119.9835715,"caller":"datalayer/data_graph.go:116","msg":"auto-created default producer","producer":"approx-prefix-cache-producer/approx-prefix-cache-producer","dataKey":"PrefixCacheMatchInfoDataKey/approx-prefix-cache-producer","consumer":"disagg-profile-handler"} [e2e-llm-inference-service] {"level":"info","ts":1785236119.9835956,"caller":"datalayer/data_graph.go:116","msg":"auto-created default producer","producer":"inflight-load-producer/inflight-load-producer","dataKey":"InFlightLoadDataKey/inflight-load-producer","consumer":"active-request-scorer"} [e2e-llm-inference-service] {"level":"info","ts":1785236119.983634,"caller":"datalayer/data_graph.go:116","msg":"auto-created default producer","producer":"token-producer/token-producer","dataKey":"TokenizedPrompt/token-producer","consumer":"disagg-profile-handler"} [e2e-llm-inference-service] {"level":"info","ts":1785236119.9837937,"caller":"runner/runner.go:685","msg":"loaded configuration from file/text successfully"} [e2e-llm-inference-service] {"level":"info","ts":1785236119.9838061,"logger":"setup","caller":"runner/runner.go:308","msg":"EPP config after phase two","config":"{SchedulerConfig:{ProfileHandler: disagg-profile-handler/disagg-profile-handler, Profiles: map[decode:{Filters: [decode-filter/by-label], Scorers: [active-request-scorer/active-request-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000], Picker: max-score-picker/max-score-picker} prefill:{Filters: [prefill-filter/by-label], Scorers: [prefix-cache-scorer/prefix-cache-scorer: 3.000000, queue-scorer/queue-scorer: 2.000000, kv-cache-utilization-scorer/kv-cache-utilization-scorer: 2.000000], Picker: max-score-picker/max-score-picker}]} SaturationDetector:0xc0005fda00 DataConfig:{Sources:[{Plugin:0xc00049b0e0 Extractors:[0xc0005fdc00]}]} FlowControlConfig: ParserRegistry:0xc000a80080}"} [e2e-llm-inference-service] {"level":"info","ts":1785236120.0034242,"logger":"setup","caller":"runner/runner.go:352","msg":"Setting pprof handlers"} [e2e-llm-inference-service] {"level":"info","ts":1785236120.0034626,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/"} [e2e-llm-inference-service] {"level":"info","ts":1785236120.003481,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/cmdline"} [e2e-llm-inference-service] {"level":"info","ts":1785236120.003486,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/trace"} [e2e-llm-inference-service] {"level":"info","ts":1785236120.0034902,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/heap"} [e2e-llm-inference-service] {"level":"info","ts":1785236120.0034947,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/goroutine"} [e2e-llm-inference-service] {"level":"info","ts":1785236120.003499,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/threadcreate"} [e2e-llm-inference-service] {"level":"info","ts":1785236120.0035036,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/profile"} [e2e-llm-inference-service] {"level":"info","ts":1785236120.0035076,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/symbol"} [e2e-llm-inference-service] {"level":"info","ts":1785236120.0035124,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/allocs"} [e2e-llm-inference-service] {"level":"info","ts":1785236120.0035164,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/block"} [e2e-llm-inference-service] {"level":"info","ts":1785236120.0035207,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/mutex"} [e2e-llm-inference-service] {"level":"info","ts":1785236120.0035284,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/plugins/state"} [e2e-llm-inference-service] {"level":"info","ts":1785236120.0035331,"logger":"setup","caller":"runner/runner.go:373","msg":"parsed config","scheduler-config":"{ProfileHandler: disagg-profile-handler/disagg-profile-handler, Profiles: map[decode:{Filters: [decode-filter/by-label], Scorers: [active-request-scorer/active-request-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000], Picker: max-score-picker/max-score-picker} prefill:{Filters: [prefill-filter/by-label], Scorers: [prefix-cache-scorer/prefix-cache-scorer: 3.000000, queue-scorer/queue-scorer: 2.000000, kv-cache-utilization-scorer/kv-cache-utilization-scorer: 2.000000], Picker: max-score-picker/max-score-picker}]}"} [e2e-llm-inference-service] {"level":"info","ts":1785236120.0035708,"logger":"setup","caller":"datalayer/runtime.go:99","msg":"Configuring datalayer runtime","numSources":1} [e2e-llm-inference-service] {"level":"info","ts":1785236120.00358,"logger":"setup","caller":"datalayer/runtime.go:118","msg":"Processing source","source":"metrics-data-source","numExtractors":1} [e2e-llm-inference-service] {"level":"info","ts":1785236120.0035946,"logger":"setup","caller":"datalayer/runtime.go:147","msg":"Source configured","source":"metrics-data-source","extractors":["core-metrics-extractor/core-metrics-extractor"]} [e2e-llm-inference-service] {"level":"info","ts":1785236120.003639,"logger":"setup","caller":"datalayer/runtime.go:206","msg":"Datalayer runtime configured","pollers":1,"notifiers":0,"endpointSources":1} [e2e-llm-inference-service] {"level":"info","ts":1785236120.0036502,"logger":"setup","caller":"runner/runner.go:833","msg":"Experimental Flow Control layer is disabled, using legacy admission control"} [e2e-llm-inference-service] {"level":"info","ts":1785236120.0037384,"logger":"setup","caller":"runner/runner.go:721","msg":"ExtProc server runner added to manager."} [e2e-llm-inference-service] {"level":"info","ts":1785236120.0037494,"logger":"setup","caller":"runner/runner.go:260","msg":"Controller manager starting"} [e2e-llm-inference-service] {"level":"info","ts":1785236120.0037777,"logger":"controller-runtime.metrics","caller":"server/server.go:208","msg":"Starting metrics server"} [e2e-llm-inference-service] {"level":"info","ts":1785236120.003966,"caller":"runnable/grpc.go:35","msg":"gRPC server starting","name":"health"} [e2e-llm-inference-service] {"level":"info","ts":1785236120.0043151,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","source":"kind source: *v1.InferencePool"} [e2e-llm-inference-service] {"level":"info","ts":1785236120.0043674,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferencemodelrewrite","controllerGroup":"llm-d.ai","controllerKind":"InferenceModelRewrite","source":"kind source: *v1alpha2.InferenceModelRewrite"} [e2e-llm-inference-service] {"level":"info","ts":1785236120.0043125,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferenceobjective","controllerGroup":"llm-d.ai","controllerKind":"InferenceObjective","source":"kind source: *v1alpha2.InferenceObjective"} [e2e-llm-inference-service] {"level":"info","ts":1785236120.004444,"caller":"runnable/grpc.go:43","msg":"gRPC server listening","name":"health","port":9003} [e2e-llm-inference-service] {"level":"info","ts":1785236120.004356,"logger":"controller-runtime.metrics","caller":"server/server.go:247","msg":"Serving metrics server","bindAddress":":9090","secure":false} [e2e-llm-inference-service] {"level":"info","ts":1785236120.0046103,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"pod","controllerGroup":"","controllerKind":"Pod","source":"kind source: *v1.Pod"} [e2e-llm-inference-service] {"level":"info","ts":1785236120.0051746,"caller":"runnable/grpc.go:35","msg":"gRPC server starting","name":"ext-proc"} [e2e-llm-inference-service] {"level":"info","ts":1785236120.0053065,"caller":"runnable/grpc.go:43","msg":"gRPC server listening","name":"ext-proc","port":9002} [e2e-llm-inference-service] {"level":"info","ts":1785236120.0098302,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.InferenceModelRewrite","reflector":"go/pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1785236120.009836,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.InferenceObjective","reflector":"go/pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1785236120.0099006,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.InferencePool","reflector":"go/pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1785236120.0105698,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Pod","reflector":"go/pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1785236120.1050756,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferencemodelrewrite","controllerGroup":"llm-d.ai","controllerKind":"InferenceModelRewrite"} [e2e-llm-inference-service] {"level":"info","ts":1785236120.1051164,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferencemodelrewrite","controllerGroup":"llm-d.ai","controllerKind":"InferenceModelRewrite","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1785236120.1051056,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferenceobjective","controllerGroup":"llm-d.ai","controllerKind":"InferenceObjective"} [e2e-llm-inference-service] {"level":"info","ts":1785236120.1051586,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferenceobjective","controllerGroup":"llm-d.ai","controllerKind":"InferenceObjective","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1785236120.2055008,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"pod","controllerGroup":"","controllerKind":"Pod"} [e2e-llm-inference-service] {"level":"info","ts":1785236120.2055354,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"pod","controllerGroup":"","controllerKind":"Pod","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1785236120.2055078,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool"} [e2e-llm-inference-service] {"level":"info","ts":1785236120.2055573,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1785236120.2057126,"caller":"controller/inferencepool_reconciler.go:46","msg":"Reconciling InferencePool","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool","namespace":"e2e-test-llm-inference-service-48639af5"},"namespace":"e2e-test-llm-inference-service-48639af5","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool","reconcileID":"ea222dfd-276b-497d-81ac-3d5d906176f7"} [e2e-llm-inference-service] {"level":"info","ts":1785236129.2058733,"caller":"controller/inferencepool_reconciler.go:46","msg":"Reconciling InferencePool","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool","namespace":"e2e-test-llm-inference-service-48639af5"},"namespace":"e2e-test-llm-inference-service-48639af5","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool","reconcileID":"6e54b79b-57d3-43aa-93fb-bfc92968e59e"} [e2e-llm-inference-service] {"level":"info","ts":1785236244.3056924,"caller":"inflightload/producer.go:308","msg":"Injected dynamic attribute into endpoint","key":"InFlightLoadDataKey/inflight-load-producer","endpoint":"e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv-rank-0"} [e2e-llm-inference-service] {"level":"info","ts":1785236244.305752,"caller":"controller/pod_reconciler.go:99","msg":"Pod already exists","controller":"pod","controllerGroup":"","controllerKind":"Pod","Pod":{"name":"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv","namespace":"e2e-test-llm-inference-service-48639af5"},"namespace":"e2e-test-llm-inference-service-48639af5","name":"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv","reconcileID":"aba4e179-44bb-4174-80c6-b27ff1ce61ae"} [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-service [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 5e20820b-f981-49f0-9c62-080ed6d7c2e9 [e2e-llm-inference-service] resourceVersion: '58605' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-28T10:55:51Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T10:55:51Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - ip: 10.133.0.63 [e2e-llm-inference-service] nodeName: ip-10-0-131-88.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche5mbz2 [e2e-llm-inference-service] uid: b32e5bc0-dc67-488e-9512-9e7bc49415d0 [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: e7d1c1d0-63a0-41df-900f-3d93b25e0eb5 [e2e-llm-inference-service] resourceVersion: '59559' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-28T10:57:24Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T10:57:24Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - ip: 10.133.0.62 [e2e-llm-inference-service] nodeName: ip-10-0-131-88.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv [e2e-llm-inference-service] uid: 1a81ee1e-7b05-4d81-a30f-b69c71d2d9d8 [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 1a81ee1e-7b05-4d81-a30f-b69c71d2d9d8 [e2e-llm-inference-service] resourceVersion: '59557' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 6d9fffb56c [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.133.0.62/23"],"mac_address":"0a:58:0a:85:00:3e","gateway_ips":["10.133.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.133.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.133.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.133.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.133.0.1"}],"ip_address":"10.133.0.62/23","gateway_ip":"10.133.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.133.0.62\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:85:00:3e\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c [e2e-llm-inference-service] uid: 74ce1bb7-3dab-4bd6-b71b-2b4ca37b3ca6 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-131-88 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"74ce1bb7-3dab-4bd6-b71b-2b4ca37b3ca6"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8001,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"llm-d-routing-sidecar"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAMESPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T10:55:24Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T10:57:24Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:initContainerStatuses: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.133.0.62"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] - name: kube-api-access-sddl4 [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/pd-sidecar [e2e-llm-inference-service] - --port=8000 [e2e-llm-inference-service] - --vllm-port=8001 [e2e-llm-inference-service] - --kv-connector=nixlv2 [e2e-llm-inference-service] - --enable-ssrf-protection=true [e2e-llm-inference-service] - --pool-group=inference.networking.x-k8s.io [e2e-llm-inference-service] - --inference-pool=e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] - --secure-proxy=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - --enable-tls=decoder [e2e-llm-inference-service] - --enable-tls=prefiller [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: INFERENCE_POOL_NAMESPACE [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] - name: INFERENCE_POOL_NAME [e2e-llm-inference-service] value: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kube-api-access-sddl4 [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000980000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to infer\ [e2e-llm-inference-service] \ RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null\n\ [e2e-llm-inference-service] \ grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/* 2>/dev/null\n\ [e2e-llm-inference-service] \n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"$hca_dir\"\ [e2e-llm-inference-service] \ ]; then\n hca_name=$(basename \"$hca_dir\")\n port_state_file=\"\ [e2e-llm-inference-service] $hca_dir/ports/1/state\" # Assume port 1\n type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\ [e2e-llm-inference-service] \n\n echo \"[Infer RoCE] Check if the port state file ${port_state_file}\ [e2e-llm-inference-service] \ exists and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] &&\ [e2e-llm-inference-service] \ grep -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found active\ [e2e-llm-inference-service] \ HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n else\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Skipping inactive or down HCA: $hca_name\"\ [e2e-llm-inference-service] \n fi\n fi\n done\n\n # Check if we found any active HCAs\n\ [e2e-llm-inference-service] \ if [ ${#active_hcas[@]} -gt 0 ]; then\n # Join the array elements with\ [e2e-llm-inference-service] \ a comma\n hca_port_pairs=()\n for hca in \"${active_hcas[@]}\";\ [e2e-llm-inference-service] \ do\n hca_port_pairs+=(\"${hca}:1\")\n done\n\n active_hca_list=$(IFS=,;\ [e2e-llm-inference-service] \ echo \"${active_hcas[*]}\")\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\"\ [e2e-llm-inference-service] )\n echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n\ [e2e-llm-inference-service] \ export UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found. NCCL_IB_HCA\ [e2e-llm-inference-service] \ will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Finding GID_INDEX for each active HCA (SR-IOV compatible)...\"\ [e2e-llm-inference-service] \n\n # For SR-IOV environments, find the most common IPv4 RoCE v2 GID index\ [e2e-llm-inference-service] \ across all HCAs\n declare -A gid_index_count\n declare -A hca_gid_index\n\ [e2e-llm-inference-service] \n for hca_name in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Processing HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for\ [e2e-llm-inference-service] \ this HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"$tpath\"\ [e2e-llm-inference-service] \ 2>/dev/null; then\n idx=$(basename \"$tpath\")\n \ [e2e-llm-inference-service] \ gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n \ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo \"\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Found IPv4 RoCE v2 GID for ${hca_name}:\ [e2e-llm-inference-service] \ index=${idx}, gid=${gid_value}\"\n hca_gid_index[\"${hca_name}\"\ [e2e-llm-inference-service] ]=\"${idx}\"\n gid_index_count[\"${idx}\"]=$((${gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]} + 1))\n break # Use first found IPv4 GID per\ [e2e-llm-inference-service] \ HCA\n fi\n fi\n done\n done\n\n\ [e2e-llm-inference-service] \ # Find the most common GID index (most likely to be consistent across\ [e2e-llm-inference-service] \ nodes)\n best_gid_index=\"\"\n max_count=0\n for idx in \"\ [e2e-llm-inference-service] ${!gid_index_count[@]}\"; do\n count=${gid_index_count[\"${idx}\"]}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n \ [e2e-llm-inference-service] \ if [ $count -gt $max_count ]; then\n max_count=$count\n\ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n #\ [e2e-llm-inference-service] \ Use deterministic fallback if tied - prefer index 3 (SR-IOV standard)\n \ [e2e-llm-inference-service] \ if [ ${#gid_index_count[@]} -gt 1 ]; then\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Multiple GID indices found, selecting most common: ${best_gid_index}\"\n \ [e2e-llm-inference-service] \ # If there's a tie, prefer index 3 as it's most common in SR-IOV setups\n\ [e2e-llm-inference-service] \ if [ -n \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\"\ [e2e-llm-inference-service] \ -eq \"$max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for NCCL,\ [e2e-llm-inference-service] \ NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR: No valid\ [e2e-llm-inference-service] \ IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any HCA.\"\n \ [e2e-llm-inference-service] \ fi\n else\n echo \"[Infer RoCE] No active HCAs found, skipping GID_INDEX\ [e2e-llm-inference-service] \ inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints landed in vLLM\ [e2e-llm-inference-service] \ 0.16.0 (vllm-project/vllm#30011).\n# Older versions still need the blanket\ [e2e-llm-inference-service] \ --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+ ]] &&\ [e2e-llm-inference-service] \ [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort -V | head\ [e2e-llm-inference-service] \ -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout 40\"\ [e2e-llm-inference-service] \nfi\n\n# --kv-transfer-config with OffloadingConnector requires vLLM 0.22.0+\ [e2e-llm-inference-service] \ (vllm-project/vllm#40020).\nKV_TRANSFER_ARGS=\"\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.22.0\" ]; then\n if [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\" != *\"\ [e2e-llm-inference-service] --kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"* ]] &&\ [e2e-llm-inference-service] \ [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n KV_TRANSFER_ARGS=\"\"\ [e2e-llm-inference-service] \n fi\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name \"\ [e2e-llm-inference-service] facebook/opt-125m\" \"publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8001 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n ${KV_TRANSFER_ARGS} \\\n \\\n --enable-ssl-refresh \\\n --ssl-certfile\ [e2e-llm-inference-service] \ /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8001 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-sddl4 [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000980000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] nodeName: ip-10-0-131-88.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c31,c25 [e2e-llm-inference-service] fsGroup: 1000980000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-s8znt [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-c2c8r [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T10:55:24Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T10:55:24Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T10:57:24Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T10:57:24Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] hostIP: 10.0.131.88 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.131.88 [e2e-llm-inference-service] podIP: 10.133.0.62 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.133.0.62 [e2e-llm-inference-service] startTime: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] initContainerStatuses: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-07-28T10:55:24Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.9.0 [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-llm-d-router-disagg-sidecar@sha256:4d57f8fe65a63b44f345487af2b831e64ddf7fb01126d4f51cc8cf49b7412d00 [e2e-llm-inference-service] containerID: cri-o://4192bf9d2ed5eae0c6ebd299775d86420492f7fedab99510fcc37def6129a8a4 [e2e-llm-inference-service] started: true [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-sddl4 [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000980000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000980000 [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-07-28T10:55:24Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: docker.io/vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] imageID: docker.io/vllm/vllm-openai-cpu@sha256:a8257c201fd2f696c146615987be61579aa2052367b73c27bd3fb1ab8e90bd4b [e2e-llm-inference-service] containerID: cri-o://63db77fcb53796bcad40f97f18680a91e18932b0ebb33c4d0ca7a640b473d15a [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-sddl4 [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000980000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000980000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b89cd7957- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 9f2f8d3c-6c27-4e8b-92c4-d22da1bb1cbe [e2e-llm-inference-service] resourceVersion: '58073' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 7b89cd7957 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.134.0.64/23"],"mac_address":"0a:58:0a:86:00:40","gateway_ips":["10.134.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.134.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.134.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.134.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.134.0.1"}],"ip_address":"10.134.0.64/23","gateway_ip":"10.134.0.1","role":"primary"}}' [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b89cd7957 [e2e-llm-inference-service] uid: fd6ae0fa-c952-42fd-8536-a098c9a5d5ae [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-141-50 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"fd6ae0fa-c952-42fd-8536-a098c9a5d5ae"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] - name: kube-api-access-8fnfl [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to infer\ [e2e-llm-inference-service] \ RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null\n\ [e2e-llm-inference-service] \ grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/* 2>/dev/null\n\ [e2e-llm-inference-service] \n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"$hca_dir\"\ [e2e-llm-inference-service] \ ]; then\n hca_name=$(basename \"$hca_dir\")\n port_state_file=\"\ [e2e-llm-inference-service] $hca_dir/ports/1/state\" # Assume port 1\n type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\ [e2e-llm-inference-service] \n\n echo \"[Infer RoCE] Check if the port state file ${port_state_file}\ [e2e-llm-inference-service] \ exists and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] &&\ [e2e-llm-inference-service] \ grep -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found active\ [e2e-llm-inference-service] \ HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n else\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Skipping inactive or down HCA: $hca_name\"\ [e2e-llm-inference-service] \n fi\n fi\n done\n\n # Check if we found any active HCAs\n\ [e2e-llm-inference-service] \ if [ ${#active_hcas[@]} -gt 0 ]; then\n # Join the array elements with\ [e2e-llm-inference-service] \ a comma\n hca_port_pairs=()\n for hca in \"${active_hcas[@]}\";\ [e2e-llm-inference-service] \ do\n hca_port_pairs+=(\"${hca}:1\")\n done\n\n active_hca_list=$(IFS=,;\ [e2e-llm-inference-service] \ echo \"${active_hcas[*]}\")\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\"\ [e2e-llm-inference-service] )\n echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n\ [e2e-llm-inference-service] \ export UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found. NCCL_IB_HCA\ [e2e-llm-inference-service] \ will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Finding GID_INDEX for each active HCA (SR-IOV compatible)...\"\ [e2e-llm-inference-service] \n\n # For SR-IOV environments, find the most common IPv4 RoCE v2 GID index\ [e2e-llm-inference-service] \ across all HCAs\n declare -A gid_index_count\n declare -A hca_gid_index\n\ [e2e-llm-inference-service] \n for hca_name in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Processing HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for\ [e2e-llm-inference-service] \ this HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"$tpath\"\ [e2e-llm-inference-service] \ 2>/dev/null; then\n idx=$(basename \"$tpath\")\n \ [e2e-llm-inference-service] \ gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n \ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo \"\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Found IPv4 RoCE v2 GID for ${hca_name}:\ [e2e-llm-inference-service] \ index=${idx}, gid=${gid_value}\"\n hca_gid_index[\"${hca_name}\"\ [e2e-llm-inference-service] ]=\"${idx}\"\n gid_index_count[\"${idx}\"]=$((${gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]} + 1))\n break # Use first found IPv4 GID per\ [e2e-llm-inference-service] \ HCA\n fi\n fi\n done\n done\n\n\ [e2e-llm-inference-service] \ # Find the most common GID index (most likely to be consistent across\ [e2e-llm-inference-service] \ nodes)\n best_gid_index=\"\"\n max_count=0\n for idx in \"\ [e2e-llm-inference-service] ${!gid_index_count[@]}\"; do\n count=${gid_index_count[\"${idx}\"]}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n \ [e2e-llm-inference-service] \ if [ $count -gt $max_count ]; then\n max_count=$count\n\ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n #\ [e2e-llm-inference-service] \ Use deterministic fallback if tied - prefer index 3 (SR-IOV standard)\n \ [e2e-llm-inference-service] \ if [ ${#gid_index_count[@]} -gt 1 ]; then\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Multiple GID indices found, selecting most common: ${best_gid_index}\"\n \ [e2e-llm-inference-service] \ # If there's a tie, prefer index 3 as it's most common in SR-IOV setups\n\ [e2e-llm-inference-service] \ if [ -n \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\"\ [e2e-llm-inference-service] \ -eq \"$max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for NCCL,\ [e2e-llm-inference-service] \ NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR: No valid\ [e2e-llm-inference-service] \ IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any HCA.\"\n \ [e2e-llm-inference-service] \ fi\n else\n echo \"[Infer RoCE] No active HCAs found, skipping GID_INDEX\ [e2e-llm-inference-service] \ inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints landed in vLLM\ [e2e-llm-inference-service] \ 0.16.0 (vllm-project/vllm#30011).\n# Older versions still need the blanket\ [e2e-llm-inference-service] \ --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+ ]] &&\ [e2e-llm-inference-service] \ [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort -V | head\ [e2e-llm-inference-service] \ -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout 40\"\ [e2e-llm-inference-service] \nfi\n\n# --kv-transfer-config with OffloadingConnector requires vLLM 0.22.0+\ [e2e-llm-inference-service] \ (vllm-project/vllm#40020).\nKV_TRANSFER_ARGS=\"\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.22.0\" ]; then\n if [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\" != *\"\ [e2e-llm-inference-service] --kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"* ]] &&\ [e2e-llm-inference-service] \ [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n KV_TRANSFER_ARGS=\"\"\ [e2e-llm-inference-service] \n fi\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name \"\ [e2e-llm-inference-service] facebook/opt-125m\" \\\n --port 8000 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n ${KV_TRANSFER_ARGS} \\\n \\\n --enable-ssl-refresh \\\n --ssl-certfile\ [e2e-llm-inference-service] \ /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-8fnfl [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: File [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000980000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: default [e2e-llm-inference-service] serviceAccount: default [e2e-llm-inference-service] nodeName: ip-10-0-141-50.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c31,c25 [e2e-llm-inference-service] fsGroup: 1000980000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-s8znt [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Pending [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] reason: ContainersNotReady [e2e-llm-inference-service] message: 'containers with unready status: [main]' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] reason: ContainersNotReady [e2e-llm-inference-service] message: 'containers with unready status: [main]' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] hostIP: 10.0.141.50 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.141.50 [e2e-llm-inference-service] startTime: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] waiting: [e2e-llm-inference-service] reason: ContainerCreating [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: false [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] imageID: '' [e2e-llm-inference-service] started: false [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-8fnfl [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche5mbz2 [e2e-llm-inference-service] generateName: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-7fc69df7b8- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: b32e5bc0-dc67-488e-9512-9e7bc49415d0 [e2e-llm-inference-service] resourceVersion: '58603' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 7fc69df7b8 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.10.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.133.0.63/23"],"mac_address":"0a:58:0a:85:00:3f","gateway_ips":["10.133.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.133.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.133.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.133.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.133.0.1"}],"ip_address":"10.133.0.63/23","gateway_ip":"10.133.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.133.0.63\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:85:00:3f\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-7fc69df7b8 [e2e-llm-inference-service] uid: ae8ede08-c725-4fec-b751-c9030c960eaf [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-131-88 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"ae8ede08-c725-4fec-b751-c9030c960eaf"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T10:55:51Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.133.0.63"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kube-api-access-5vpmt [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: llm-d.ai/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n- type:\ [e2e-llm-inference-service] \ disagg-headers-handler\n- type: prefill-filter\n- type: decode-filter\n- type:\ [e2e-llm-inference-service] \ queue-scorer\n- type: kv-cache-utilization-scorer\n- type: active-request-scorer\n\ [e2e-llm-inference-service] - type: prefix-cache-scorer\n- type: max-score-picker\n- type: always-disagg-pd-decider\n\ [e2e-llm-inference-service] - parameters:\n deciders:\n prefill: always-disagg-pd-decider\n type:\ [e2e-llm-inference-service] \ disagg-profile-handler\n- parameters:\n scheme: https\n type: metrics-data-source\n\ [e2e-llm-inference-service] schedulingProfiles:\n- name: prefill\n plugins:\n - pluginRef: prefill-filter\n\ [e2e-llm-inference-service] \ - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: queue-scorer\n\ [e2e-llm-inference-service] \ weight: 2\n - pluginRef: kv-cache-utilization-scorer\n weight: 2\n\ [e2e-llm-inference-service] \ - pluginRef: max-score-picker\n- name: decode\n plugins:\n - pluginRef:\ [e2e-llm-inference-service] \ decode-filter\n - pluginRef: active-request-scorer\n weight: 2\n - pluginRef:\ [e2e-llm-inference-service] \ prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kube-api-access-5vpmt [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000980000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] nodeName: ip-10-0-131-88.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c31,c25 [e2e-llm-inference-service] fsGroup: 1000980000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-s8znt [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-cpxm2 [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T10:55:20Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T10:55:51Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T10:55:51Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] hostIP: 10.0.131.88 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.131.88 [e2e-llm-inference-service] podIP: 10.133.0.63 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.133.0.63 [e2e-llm-inference-service] startTime: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-llm-d-router-endpoint-picker@sha256:7fd1d61cb5505e026eebab0b9653758779c3dee241e37bea92a0bab961a48c50 [e2e-llm-inference-service] containerID: cri-o://1bce47f7167c065cad94715e9b69de044901dc43db5a21ae522d6658ea3364d9 [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-5vpmt [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000980000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000980000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 9648e94a-ee59-45eb-b87a-e1db2c52ac89 [e2e-llm-inference-service] resourceVersion: '58059' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] openshift.io/internal-registry-pull-secret-ref: llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-cpxm2 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: a31a8e8c-0db4-4e00-98bc-e1edb1da6f1e [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: openshift.io/image-registry-pull-secrets_service-account-controller [e2e-llm-inference-service] operation: Apply [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:imagePullSecrets: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:openshift.io/internal-registry-pull-secret-ref: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] k:{"name":"llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-cpxm2"}: {} [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"a31a8e8c-0db4-4e00-98bc-e1edb1da6f1e"}: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"default-dockercfg-s8znt"}: {} [e2e-llm-inference-service] k:{"name":"seaweedfs-s3-creds"}: {} [e2e-llm-inference-service] secrets: [e2e-llm-inference-service] - name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: default-dockercfg-s8znt [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-cpxm2 [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-s8znt [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-cpxm2 [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: ServiceAccount [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 9a63aa0c-a0b2-48c6-832c-b9dcdfd164b8 [e2e-llm-inference-service] resourceVersion: '58024' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T10:55:18Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] openshift.io/internal-registry-pull-secret-ref: llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-c2c8r [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: a31a8e8c-0db4-4e00-98bc-e1edb1da6f1e [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: openshift.io/image-registry-pull-secrets_service-account-controller [e2e-llm-inference-service] operation: Apply [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T10:55:18Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:imagePullSecrets: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:openshift.io/internal-registry-pull-secret-ref: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] k:{"name":"llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-c2c8r"}: {} [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T10:55:18Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"a31a8e8c-0db4-4e00-98bc-e1edb1da6f1e"}: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"default-dockercfg-s8znt"}: {} [e2e-llm-inference-service] k:{"name":"seaweedfs-s3-creds"}: {} [e2e-llm-inference-service] secrets: [e2e-llm-inference-service] - name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: default-dockercfg-s8znt [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-c2c8r [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-s8znt [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-c2c8r [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: ServiceAccount [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-service [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 65639054-45ff-487b-9455-890dbd9b85b9 [e2e-llm-inference-service] resourceVersion: '58102' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: a31a8e8c-0db4-4e00-98bc-e1edb1da6f1e [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"a31a8e8c-0db4-4e00-98bc-e1edb1da6f1e"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] targetPort: grpc [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] targetPort: grpc-health [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] targetPort: metrics [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] targetPort: zmq [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] clusterIP: 172.31.113.187 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.113.187 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: e15c7557-b486-492e-9fec-d53425d3c688 [e2e-llm-inference-service] resourceVersion: '58046' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: a31a8e8c-0db4-4e00-98bc-e1edb1da6f1e [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"a31a8e8c-0db4-4e00-98bc-e1edb1da6f1e"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] targetPort: 8000 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] clusterIP: 172.31.159.25 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.159.25 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: c8be1023-5cb7-49da-89a1-66d824744c8d [e2e-llm-inference-service] resourceVersion: '59563' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: a31a8e8c-0db4-4e00-98bc-e1edb1da6f1e [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"a31a8e8c-0db4-4e00-98bc-e1edb1da6f1e"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:rollingUpdate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:maxSurge: {} [e2e-llm-inference-service] f:maxUnavailable: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8001,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"llm-d-routing-sidecar"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAMESPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T10:57:24Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/pd-sidecar [e2e-llm-inference-service] - --port=8000 [e2e-llm-inference-service] - --vllm-port=8001 [e2e-llm-inference-service] - --kv-connector=nixlv2 [e2e-llm-inference-service] - --enable-ssrf-protection=true [e2e-llm-inference-service] - --pool-group=inference.networking.x-k8s.io [e2e-llm-inference-service] - --inference-pool=e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] - --secure-proxy=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - --enable-tls=decoder [e2e-llm-inference-service] - --enable-tls=prefiller [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: INFERENCE_POOL_NAMESPACE [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] - name: INFERENCE_POOL_NAME [e2e-llm-inference-service] value: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n # Check if\ [e2e-llm-inference-service] \ we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ # Join the array elements with a comma\n hca_port_pairs=()\n \ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n hca_port_pairs+=(\"\ [e2e-llm-inference-service] ${hca}:1\")\n done\n\n active_hca_list=$(IFS=,; echo \"${active_hcas[*]}\"\ [e2e-llm-inference-service] )\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\n# --kv-transfer-config with OffloadingConnector requires vLLM\ [e2e-llm-inference-service] \ 0.22.0+ (vllm-project/vllm#40020).\nKV_TRANSFER_ARGS=\"\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.22.0\" ]; then\n if [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\" !=\ [e2e-llm-inference-service] \ *\"--kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"\ [e2e-llm-inference-service] * ]] && [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n KV_TRANSFER_ARGS=\"\ [e2e-llm-inference-service] \"\n fi\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \"publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8001 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n ${KV_TRANSFER_ARGS} \\\n \\\n --enable-ssl-refresh \\\n --ssl-certfile\ [e2e-llm-inference-service] \ /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8001 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: RollingUpdate [e2e-llm-inference-service] rollingUpdate: [e2e-llm-inference-service] maxUnavailable: 25% [e2e-llm-inference-service] maxSurge: 25% [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-28T10:57:24Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T10:57:24Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-28T10:57:24Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c" [e2e-llm-inference-service] has successfully progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: d1243516-b14c-4a1e-b26b-43b6bcd22821 [e2e-llm-inference-service] resourceVersion: '64524' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: a31a8e8c-0db4-4e00-98bc-e1edb1da6f1e [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"a31a8e8c-0db4-4e00-98bc-e1edb1da6f1e"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:rollingUpdate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:maxSurge: {} [e2e-llm-inference-service] f:maxUnavailable: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T11:05:20Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:unavailableReplicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n # Check if\ [e2e-llm-inference-service] \ we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ # Join the array elements with a comma\n hca_port_pairs=()\n \ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n hca_port_pairs+=(\"\ [e2e-llm-inference-service] ${hca}:1\")\n done\n\n active_hca_list=$(IFS=,; echo \"${active_hcas[*]}\"\ [e2e-llm-inference-service] )\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\n# --kv-transfer-config with OffloadingConnector requires vLLM\ [e2e-llm-inference-service] \ 0.22.0+ (vllm-project/vllm#40020).\nKV_TRANSFER_ARGS=\"\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.22.0\" ]; then\n if [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\" !=\ [e2e-llm-inference-service] \ *\"--kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"\ [e2e-llm-inference-service] * ]] && [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n KV_TRANSFER_ARGS=\"\ [e2e-llm-inference-service] \"\n fi\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \\\n --port 8000 \\\n ${ACCESS_LOG_ARGS} \\\n\ [e2e-llm-inference-service] \ ${SHUTDOWN_TIMEOUT_ARGS} \\\n ${KV_TRANSFER_ARGS} \\\n \\\n --enable-ssl-refresh\ [e2e-llm-inference-service] \ \\\n --ssl-certfile /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: File [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: RollingUpdate [e2e-llm-inference-service] rollingUpdate: [e2e-llm-inference-service] maxUnavailable: 25% [e2e-llm-inference-service] maxSurge: 25% [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] unavailableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastUpdateTime: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastUpdateTime: '2026-07-28T11:05:20Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T11:05:20Z' [e2e-llm-inference-service] reason: ProgressDeadlineExceeded [e2e-llm-inference-service] message: ReplicaSet "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b89cd7957" [e2e-llm-inference-service] has timed out progressing. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: b5b24367-157d-4bba-9057-7c81c3345901 [e2e-llm-inference-service] resourceVersion: '58607' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: a31a8e8c-0db4-4e00-98bc-e1edb1da6f1e [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"a31a8e8c-0db4-4e00-98bc-e1edb1da6f1e"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T10:55:51Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.10.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: llm-d.ai/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n-\ [e2e-llm-inference-service] \ type: disagg-headers-handler\n- type: prefill-filter\n- type: decode-filter\n\ [e2e-llm-inference-service] - type: queue-scorer\n- type: kv-cache-utilization-scorer\n- type: active-request-scorer\n\ [e2e-llm-inference-service] - type: prefix-cache-scorer\n- type: max-score-picker\n- type: always-disagg-pd-decider\n\ [e2e-llm-inference-service] - parameters:\n deciders:\n prefill: always-disagg-pd-decider\n\ [e2e-llm-inference-service] \ type: disagg-profile-handler\n- parameters:\n scheme: https\n type:\ [e2e-llm-inference-service] \ metrics-data-source\nschedulingProfiles:\n- name: prefill\n plugins:\n\ [e2e-llm-inference-service] \ - pluginRef: prefill-filter\n - pluginRef: prefix-cache-scorer\n \ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: queue-scorer\n weight: 2\n - pluginRef:\ [e2e-llm-inference-service] \ kv-cache-utilization-scorer\n weight: 2\n - pluginRef: max-score-picker\n\ [e2e-llm-inference-service] - name: decode\n plugins:\n - pluginRef: decode-filter\n - pluginRef:\ [e2e-llm-inference-service] \ active-request-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n\ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: Recreate [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-28T10:55:51Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T10:55:51Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-28T10:55:51Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-7fc69df7b8" [e2e-llm-inference-service] has successfully progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 74ce1bb7-3dab-4bd6-b71b-2b4ca37b3ca6 [e2e-llm-inference-service] resourceVersion: '59562' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 6d9fffb56c [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '2' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] uid: c8be1023-5cb7-49da-89a1-66d824744c8d [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"c8be1023-5cb7-49da-89a1-66d824744c8d"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8001,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"llm-d-routing-sidecar"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAMESPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T10:57:24Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 6d9fffb56c [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 6d9fffb56c [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/pd-sidecar [e2e-llm-inference-service] - --port=8000 [e2e-llm-inference-service] - --vllm-port=8001 [e2e-llm-inference-service] - --kv-connector=nixlv2 [e2e-llm-inference-service] - --enable-ssrf-protection=true [e2e-llm-inference-service] - --pool-group=inference.networking.x-k8s.io [e2e-llm-inference-service] - --inference-pool=e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] - --secure-proxy=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - --enable-tls=decoder [e2e-llm-inference-service] - --enable-tls=prefiller [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: INFERENCE_POOL_NAMESPACE [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] - name: INFERENCE_POOL_NAME [e2e-llm-inference-service] value: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n # Check if\ [e2e-llm-inference-service] \ we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ # Join the array elements with a comma\n hca_port_pairs=()\n \ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n hca_port_pairs+=(\"\ [e2e-llm-inference-service] ${hca}:1\")\n done\n\n active_hca_list=$(IFS=,; echo \"${active_hcas[*]}\"\ [e2e-llm-inference-service] )\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\n# --kv-transfer-config with OffloadingConnector requires vLLM\ [e2e-llm-inference-service] \ 0.22.0+ (vllm-project/vllm#40020).\nKV_TRANSFER_ARGS=\"\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.22.0\" ]; then\n if [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\" !=\ [e2e-llm-inference-service] \ *\"--kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"\ [e2e-llm-inference-service] * ]] && [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n KV_TRANSFER_ARGS=\"\ [e2e-llm-inference-service] \"\n fi\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \"publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8001 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n ${KV_TRANSFER_ARGS} \\\n \\\n --enable-ssl-refresh \\\n --ssl-certfile\ [e2e-llm-inference-service] \ /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8001 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b89cd7957 [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: fd6ae0fa-c952-42fd-8536-a098c9a5d5ae [e2e-llm-inference-service] resourceVersion: '58070' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 7b89cd7957 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '2' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill [e2e-llm-inference-service] uid: d1243516-b14c-4a1e-b26b-43b6bcd22821 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d1243516-b14c-4a1e-b26b-43b6bcd22821"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 7b89cd7957 [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 7b89cd7957 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n # Check if\ [e2e-llm-inference-service] \ we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ # Join the array elements with a comma\n hca_port_pairs=()\n \ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n hca_port_pairs+=(\"\ [e2e-llm-inference-service] ${hca}:1\")\n done\n\n active_hca_list=$(IFS=,; echo \"${active_hcas[*]}\"\ [e2e-llm-inference-service] )\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\n# --kv-transfer-config with OffloadingConnector requires vLLM\ [e2e-llm-inference-service] \ 0.22.0+ (vllm-project/vllm#40020).\nKV_TRANSFER_ARGS=\"\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.22.0\" ]; then\n if [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\" !=\ [e2e-llm-inference-service] \ *\"--kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"\ [e2e-llm-inference-service] * ]] && [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n KV_TRANSFER_ARGS=\"\ [e2e-llm-inference-service] \"\n fi\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \\\n --port 8000 \\\n ${ACCESS_LOG_ARGS} \\\n\ [e2e-llm-inference-service] \ ${SHUTDOWN_TIMEOUT_ARGS} \\\n ${KV_TRANSFER_ARGS} \\\n \\\n --enable-ssl-refresh\ [e2e-llm-inference-service] \ \\\n --ssl-certfile /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: File [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-7fc69df7b8 [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: ae8ede08-c725-4fec-b751-c9030c960eaf [e2e-llm-inference-service] resourceVersion: '58606' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 7fc69df7b8 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler [e2e-llm-inference-service] uid: b5b24367-157d-4bba-9057-7c81c3345901 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b5b24367-157d-4bba-9057-7c81c3345901"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T10:55:51Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 7fc69df7b8 [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 7fc69df7b8 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.10.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: llm-d.ai/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n-\ [e2e-llm-inference-service] \ type: disagg-headers-handler\n- type: prefill-filter\n- type: decode-filter\n\ [e2e-llm-inference-service] - type: queue-scorer\n- type: kv-cache-utilization-scorer\n- type: active-request-scorer\n\ [e2e-llm-inference-service] - type: prefix-cache-scorer\n- type: max-score-picker\n- type: always-disagg-pd-decider\n\ [e2e-llm-inference-service] - parameters:\n deciders:\n prefill: always-disagg-pd-decider\n\ [e2e-llm-inference-service] \ type: disagg-profile-handler\n- parameters:\n scheme: https\n type:\ [e2e-llm-inference-service] \ metrics-data-source\nschedulingProfiles:\n- name: prefill\n plugins:\n\ [e2e-llm-inference-service] \ - pluginRef: prefill-filter\n - pluginRef: prefix-cache-scorer\n \ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: queue-scorer\n weight: 2\n - pluginRef:\ [e2e-llm-inference-service] \ kv-cache-utilization-scorer\n weight: 2\n - pluginRef: max-score-picker\n\ [e2e-llm-inference-service] - name: decode\n plugins:\n - pluginRef: decode-filter\n - pluginRef:\ [e2e-llm-inference-service] \ active-request-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n\ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-rb [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 53fedc7f-5d12-4eb8-b096-8efe7c3b91a4 [e2e-llm-inference-service] resourceVersion: '58084' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: a31a8e8c-0db4-4e00-98bc-e1edb1da6f1e [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"a31a8e8c-0db4-4e00-98bc-e1edb1da6f1e"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] apiGroup: rbac.authorization.k8s.io [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-role [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-rb [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: b56fea26-6754-483e-a12b-c8849203fbe9 [e2e-llm-inference-service] resourceVersion: '58032' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: a31a8e8c-0db4-4e00-98bc-e1edb1da6f1e [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T10:55:18Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"a31a8e8c-0db4-4e00-98bc-e1edb1da6f1e"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] apiGroup: rbac.authorization.k8s.io [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-role [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-role [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 7997e15e-17b9-4290-acf2-f6687ef53d86 [e2e-llm-inference-service] resourceVersion: '58080' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: a31a8e8c-0db4-4e00-98bc-e1edb1da6f1e [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"a31a8e8c-0db4-4e00-98bc-e1edb1da6f1e"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - llm-d.ai [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] - create [e2e-llm-inference-service] - update [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - delete [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-role [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 53f2805c-e2c9-4170-ba48-e08cca01bee4 [e2e-llm-inference-service] resourceVersion: '58030' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T10:55:18Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: a31a8e8c-0db4-4e00-98bc-e1edb1da6f1e [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T10:55:18Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"a31a8e8c-0db4-4e00-98bc-e1edb1da6f1e"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-service-jl65l [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-e8706282-epp-service- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: ea8a46af-3a61-4497-a519-a22cbb29bd8d [e2e-llm-inference-service] resourceVersion: '58604' [e2e-llm-inference-service] generation: 3 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: llmisvc-model-pvc-router-manage-e8706282-epp-service [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-28T10:55:51Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-service [e2e-llm-inference-service] uid: 65639054-45ff-487b-9455-890dbd9b85b9 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T10:55:51Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"65639054-45ff-487b-9455-890dbd9b85b9"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.133.0.63 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche5mbz2 [e2e-llm-inference-service] uid: b32e5bc0-dc67-488e-9512-9e7bc49415d0 [e2e-llm-inference-service] nodeName: ip-10-0-131-88.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-sn55cl [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: ebeb5114-ca5f-4b36-b7e0-233cd9184e62 [e2e-llm-inference-service] resourceVersion: '59560' [e2e-llm-inference-service] generation: 3 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-28T10:57:24Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] uid: e15c7557-b486-492e-9fec-d53425d3c688 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T10:57:24Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"e15c7557-b486-492e-9fec-d53425d3c688"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.133.0.62 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv [e2e-llm-inference-service] uid: 1a81ee1e-7b05-4d81-a30f-b69c71d2d9d8 [e2e-llm-inference-service] nodeName: ip-10-0-131-88.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-rb [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 53fedc7f-5d12-4eb8-b096-8efe7c3b91a4 [e2e-llm-inference-service] resourceVersion: '58084' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: a31a8e8c-0db4-4e00-98bc-e1edb1da6f1e [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"a31a8e8c-0db4-4e00-98bc-e1edb1da6f1e"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] userNames: [e2e-llm-inference-service] - system:serviceaccount:e2e-test-llm-inference-service-48639af5:llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] groupNames: null [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-role [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-rb [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: b56fea26-6754-483e-a12b-c8849203fbe9 [e2e-llm-inference-service] resourceVersion: '58032' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: a31a8e8c-0db4-4e00-98bc-e1edb1da6f1e [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T10:55:18Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"a31a8e8c-0db4-4e00-98bc-e1edb1da6f1e"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] userNames: [e2e-llm-inference-service] - system:serviceaccount:e2e-test-llm-inference-service-48639af5:llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] groupNames: null [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-role [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-role [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 7997e15e-17b9-4290-acf2-f6687ef53d86 [e2e-llm-inference-service] resourceVersion: '58080' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: a31a8e8c-0db4-4e00-98bc-e1edb1da6f1e [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"a31a8e8c-0db4-4e00-98bc-e1edb1da6f1e"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - llm-d.ai [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - create [e2e-llm-inference-service] - delete [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - update [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-role [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 53f2805c-e2c9-4170-ba48-e08cca01bee4 [e2e-llm-inference-service] resourceVersion: '58030' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T10:55:18Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: a31a8e8c-0db4-4e00-98bc-e1edb1da6f1e [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T10:55:18Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"a31a8e8c-0db4-4e00-98bc-e1edb1da6f1e"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"a31a8e8c-0db4-4e00-98bc-e1edb1da6f1e"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T10:55:29Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-28T10:55:29Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-28T10:55:29Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-route [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: a31a8e8c-0db4-4e00-98bc-e1edb1da6f1e [e2e-llm-inference-service] resourceVersion: '58370' [e2e-llm-inference-service] uid: a3fdebb2-a904-4e3c-97ca-cabce1483195 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T10:55:29Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T10:55:20Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"a31a8e8c-0db4-4e00-98bc-e1edb1da6f1e"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T10:55:29Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-28T10:55:29Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-28T10:55:29Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-route [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: a31a8e8c-0db4-4e00-98bc-e1edb1da6f1e [e2e-llm-inference-service] resourceVersion: '58370' [e2e-llm-inference-service] uid: a3fdebb2-a904-4e3c-97ca-cabce1483195 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T10:55:29Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T10:55:20Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"a31a8e8c-0db4-4e00-98bc-e1edb1da6f1e"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:endpointPickerRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:number: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:matchLabels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPorts: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-28T10:55:29Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: a31a8e8c-0db4-4e00-98bc-e1edb1da6f1e [e2e-llm-inference-service] resourceVersion: '58353' [e2e-llm-inference-service] uid: 9ff58f17-6177-406f-88fc-7ec1ab6115be [e2e-llm-inference-service] spec: [e2e-llm-inference-service] appProtocol: http [e2e-llm-inference-service] endpointPickerRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-service [e2e-llm-inference-service] port: [e2e-llm-inference-service] number: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPorts: [e2e-llm-inference-service] - number: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T10:55:29Z' [e2e-llm-inference-service] message: Referenced by an HTTPRoute accepted by the parentRef Gateway [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T10:55:29Z' [e2e-llm-inference-service] message: Referenced ExtensionRef resolved successfully [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: networking.istio.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] kind: AuthPolicy [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T10:55:21Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-policies [e2e-llm-inference-service] app.kubernetes.io/managed-by: odh-model-controller [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"a31a8e8c-0db4-4e00-98bc-e1edb1da6f1e"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:rules: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:authentication: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:public: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:anonymous: {} [e2e-llm-inference-service] f:credentials: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:overrides: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fairness: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:response: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:success: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:headers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:x-gateway-inference-fairness-id: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:x-gateway-inference-objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:targetRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T10:55:21Z' [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Accepted"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Enforced"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-28T10:55:22Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-route-authn [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: a31a8e8c-0db4-4e00-98bc-e1edb1da6f1e [e2e-llm-inference-service] resourceVersion: '58270' [e2e-llm-inference-service] uid: 826fdd96-fc18-49f6-99fb-d6a000362b1c [e2e-llm-inference-service] spec: [e2e-llm-inference-service] rules: [e2e-llm-inference-service] authentication: [e2e-llm-inference-service] public: [e2e-llm-inference-service] anonymous: {} [e2e-llm-inference-service] credentials: {} [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] overrides: [e2e-llm-inference-service] fairness: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] objective: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] response: [e2e-llm-inference-service] success: [e2e-llm-inference-service] headers: [e2e-llm-inference-service] x-gateway-inference-fairness-id: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.fairness [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] x-gateway-inference-objective: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.objective [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-route [e2e-llm-inference-service] status: [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T10:55:21Z' [e2e-llm-inference-service] message: AuthPolicy has been accepted [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T10:55:22Z' [e2e-llm-inference-service] message: AuthPolicy has been successfully enforced [e2e-llm-inference-service] reason: Enforced [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Enforced [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"a31a8e8c-0db4-4e00-98bc-e1edb1da6f1e"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-scheduler [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: a31a8e8c-0db4-4e00-98bc-e1edb1da6f1e [e2e-llm-inference-service] resourceVersion: '58117' [e2e-llm-inference-service] uid: b0d89b81-be84-4699-bbe5-9b13f510ea25 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-epp-service.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-epp-service.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T10:55:29Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"a31a8e8c-0db4-4e00-98bc-e1edb1da6f1e"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T10:55:29Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-shadow-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: a31a8e8c-0db4-4e00-98bc-e1edb1da6f1e [e2e-llm-inference-service] resourceVersion: '58364' [e2e-llm-inference-service] uid: bf2f1506-d95f-4606-9ba5-10ca179bc510 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-inference--ip-8bbb5aea.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T10:55:20Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"a31a8e8c-0db4-4e00-98bc-e1edb1da6f1e"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T10:55:20Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: a31a8e8c-0db4-4e00-98bc-e1edb1da6f1e [e2e-llm-inference-service] resourceVersion: '58168' [e2e-llm-inference-service] uid: 07b09082-d806-4a53-bdc4-5ccf2d4be2dd [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"a31a8e8c-0db4-4e00-98bc-e1edb1da6f1e"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-scheduler [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: a31a8e8c-0db4-4e00-98bc-e1edb1da6f1e [e2e-llm-inference-service] resourceVersion: '58117' [e2e-llm-inference-service] uid: b0d89b81-be84-4699-bbe5-9b13f510ea25 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-epp-service.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-epp-service.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T10:55:29Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"a31a8e8c-0db4-4e00-98bc-e1edb1da6f1e"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T10:55:29Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-shadow-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: a31a8e8c-0db4-4e00-98bc-e1edb1da6f1e [e2e-llm-inference-service] resourceVersion: '58364' [e2e-llm-inference-service] uid: bf2f1506-d95f-4606-9ba5-10ca179bc510 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-inference--ip-8bbb5aea.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T10:55:20Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"a31a8e8c-0db4-4e00-98bc-e1edb1da6f1e"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T10:55:20Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: a31a8e8c-0db4-4e00-98bc-e1edb1da6f1e [e2e-llm-inference-service] resourceVersion: '58168' [e2e-llm-inference-service] uid: 07b09082-d806-4a53-bdc4-5ccf2d4be2dd [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"a31a8e8c-0db4-4e00-98bc-e1edb1da6f1e"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-scheduler [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: a31a8e8c-0db4-4e00-98bc-e1edb1da6f1e [e2e-llm-inference-service] resourceVersion: '58117' [e2e-llm-inference-service] uid: b0d89b81-be84-4699-bbe5-9b13f510ea25 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-epp-service.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-epp-service.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T10:55:29Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"a31a8e8c-0db4-4e00-98bc-e1edb1da6f1e"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T10:55:29Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-shadow-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: a31a8e8c-0db4-4e00-98bc-e1edb1da6f1e [e2e-llm-inference-service] resourceVersion: '58364' [e2e-llm-inference-service] uid: bf2f1506-d95f-4606-9ba5-10ca179bc510 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-inference--ip-8bbb5aea.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T10:55:20Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"a31a8e8c-0db4-4e00-98bc-e1edb1da6f1e"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T10:55:20Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: a31a8e8c-0db4-4e00-98bc-e1edb1da6f1e [e2e-llm-inference-service] resourceVersion: '58168' [e2e-llm-inference-service] uid: 07b09082-d806-4a53-bdc4-5ccf2d4be2dd [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"a31a8e8c-0db4-4e00-98bc-e1edb1da6f1e"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:extensionRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:portNumber: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPortNumber: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T10:55:19Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: a31a8e8c-0db4-4e00-98bc-e1edb1da6f1e [e2e-llm-inference-service] resourceVersion: '58107' [e2e-llm-inference-service] uid: 2d3bdbe5-d664-4c6b-a58e-99770127482a [e2e-llm-inference-service] spec: [e2e-llm-inference-service] extensionRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-service [e2e-llm-inference-service] portNumber: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPortNumber: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parent: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '1970-01-01T00:00:00Z' [e2e-llm-inference-service] message: Waiting for controller [e2e-llm-inference-service] reason: Pending [e2e-llm-inference-service] status: Unknown [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Status [e2e-llm-inference-service] name: default [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56ckd6sv [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T11:10:07Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 6d9fffb56c [e2e-llm-inference-service] timestamp: '2026-07-28T11:09:47Z' [e2e-llm-inference-service] window: 18.557s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 100384645n [e2e-llm-inference-service] memory: 2414548Ki [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 15728943n [e2e-llm-inference-service] memory: 25208Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche5mbz2 [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T11:10:07Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 7fc69df7b8 [e2e-llm-inference-service] timestamp: '2026-07-28T11:09:39Z' [e2e-llm-inference-service] window: 12.693s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 56855195n [e2e-llm-inference-service] memory: 35116Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_inference_service] [2026-07-28T11:10:08.243772] end - ❌ 902.141s: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T10:55:30Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T10:55:30Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T10:57:33Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T10:55:30Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-28T10:55:21Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T10:55:30Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T10:55:52Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T10:55:52Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T10:55:30Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ---------------------------- Captured log teardown ----------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.conftest:conftest.py:168 Skipping deletion of namespace e2e-test-llm-inference-service-48639af5 (SKIP_DELETION_ON_FAILURE) [e2e-llm-inference-service] _ test_llm_autoscaling_hpa_lws[router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-hpa] _ [e2e-llm-inference-service] [gw0] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-lws', 'prometheus-scrape', 'scaling-hpa'], prompt='KSer... {'name': 'scaling-hpa-autoscale-hpa-lws-b344a3ff'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.autoscaling_hpa [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator-lws", [e2e-llm-inference-service] "prometheus-scrape", [e2e-llm-inference-service] "scaling-hpa", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="autoscale-hpa-lws", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_multi_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_autoscaling_hpa_lws(test_case: TestCase): [e2e-llm-inference-service] """HPA + LWS: HPA exists with WVA annotations; pods scale under load.""" [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] kserve_client = _new_kserve_client() [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] ns = test_case.namespace [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > _create_and_wait(kserve_client, test_case) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:670: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-lws', 'prometheus-scrape', 'scaling-hpa'], prompt='KSer... {'name': 'scaling-hpa-autoscale-hpa-lws-b344a3ff'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] def _create_and_wait(kserve_client, test_case): [e2e-llm-inference-service] """Create LLMISVC and wait for it to be ready.""" [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:482: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...ale-hpa-b29acdba'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-lws-b344a3ff'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-07-28T11:05:25.939785', start_time = 1785236725.9400434 [e2e-llm-inference-service] duration = 900.5270466804504, timestamp_end = '2026-07-28T11:20:26.467092' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....autoscale-hpa-b29acdba'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-lws-b344a3ff'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1376: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7ff792df4d60> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1387: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T11:06:19Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T11:06:19Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T11:05:48Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T11:06:36Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T11:06:36Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T11:06:19Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T11:06:36Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T11:06:19Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T11:06:19Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1371: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:82 Created test namespace e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret seaweedfs-s3-creds from kserve-ci-e2e-test to e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret storage-config from kserve-ci-e2e-test to e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:178 Patched default SA in e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 with secret seaweedfs-s3-creds [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:162 ConfigMap odh-kserve-custom-ca-bundle already exists in e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:159 Copied ConfigMap odh-trusted-ca-bundle from kserve-ci-e2e-test to e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig router-managed-autoscale-hpa-lw-1aa98714 in namespace e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig router-managed-autoscale-hpa-lw-1aa98714 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig router-managed-autoscale-hpa-lw-1aa98714 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig workload-llmd-simulator-lws-aut-fe7a55cc in namespace e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig workload-llmd-simulator-lws-aut-fe7a55cc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig workload-llmd-simulator-lws-aut-fe7a55cc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig prometheus-scrape-autoscale-hpa-b29acdba in namespace e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig prometheus-scrape-autoscale-hpa-b29acdba [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig prometheus-scrape-autoscale-hpa-b29acdba [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig scaling-hpa-autoscale-hpa-lws-b344a3ff in namespace e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig scaling-hpa-autoscale-hpa-lws-b344a3ff [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig scaling-hpa-autoscale-hpa-lws-b344a3ff [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_autoscaling_hpa_lws] [2026-07-28T11:05:25.566522] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-llmd-simulator-lws', 'prometheus-scrape', 'scaling-hpa'], prompt='KServe is a', service_name='autoscale-hpa-lws', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, namespace='e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2', before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-lws', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-lw-1aa98714'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-lws-aut-fe7a55cc'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-b29acdba'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-lws-b344a3ff'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-28T11:05:25.580270] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-lws', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-lw-1aa98714'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-lws-aut-fe7a55cc'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-b29acdba'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-lws-b344a3ff'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-28T11:05:25.939643] end - ✅ in 0.359s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-28T11:05:25.939785] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-lws', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-lw-1aa98714'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-lws-aut-fe7a55cc'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-b29acdba'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-lws-b344a3ff'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'RouterReady', 'WorkloadsReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T11:05:48Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T11:05:48Z', 'message': 'Inference Pool e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T11:05:48Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T11:05:48Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T11:05:48Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T11:05:48Z', 'message': 'HPA conditions not yet available', 'reason': 'HPAProgressing', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T11:05:48Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T11:05:48Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T11:05:48Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'RouterReady', 'WorkloadsReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T11:06:19Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T11:06:19Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T11:05:48Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T11:06:19Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T11:06:19Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T11:06:19Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T11:06:19Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T11:06:19Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T11:06:19Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T11:06:19Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T11:06:19Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T11:05:48Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T11:06:36Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T11:06:36Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T11:06:19Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T11:06:36Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T11:06:19Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T11:06:19Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 9: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-28T10:22:24Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 10: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-28T10:22:24Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 11: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-28T10:22:24Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1391 Timed out waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T11:06:19Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T11:06:19Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T11:05:48Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T11:06:36Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T11:06:36Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T11:06:19Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T11:06:36Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T11:06:19Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T11:06:19Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-07-28T11:20:26.467092] end - ❌ 900.527s: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T11:06:19Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T11:06:19Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T11:05:48Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T11:06:36Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T11:06:36Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T11:06:19Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T11:06:36Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T11:06:19Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T11:06:19Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [delete_llmisvc] [2026-07-28T11:20:26.467176] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-lws', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-lw-1aa98714'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-lws-aut-fe7a55cc'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-b29acdba'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-lws-b344a3ff'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 3 pod(s) for autoscale-hpa-lws still terminating: ['autoscale-hpa-lws-kserve-mn-0', 'autoscale-hpa-lws-kserve-mn-0-1', 'autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-pxkr7'] [e2e-llm-inference-service] assert not ['autoscale-hpa-lws-kserve-mn-0', 'autoscale-hpa-lws-kserve-mn-0-1', 'autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-pxkr7'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 1 pod(s) for autoscale-hpa-lws still terminating: ['autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-pxkr7'] [e2e-llm-inference-service] assert not ['autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-pxkr7'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [delete_llmisvc] [2026-07-28T11:21:01.768870] end - ✅ in 35.301s [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_autoscaling_hpa_lws] [2026-07-28T11:21:01.768992] end - ❌ 936.202s: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T11:06:19Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T11:06:19Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T11:05:48Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T11:06:36Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T11:06:36Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T11:06:19Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T11:06:36Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T11:06:19Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T11:06:19Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ---------------------------- Captured log teardown ----------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.conftest:conftest.py:168 Skipping deletion of namespace e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 (SKIP_DELETION_ON_FAILURE) [e2e-llm-inference-service] _ test_llm_inference_service[router-managed-workload-simulated-dp-ep-cpu-model-pvc] _ [e2e-llm-inference-service] [gw1] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-simulated-dp-ep-cpu', 'model-pvc'], prompt='KServe is a', service_name... {'name': 'model-pvc-llmisvc-model-pvc-rou-cfc8d654'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-gateway-ref", [e2e-llm-inference-service] "router-with-managed-route", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] expected_gateway="router-gateway-1", [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda tc: create_router_resources( [e2e-llm-inference-service] gateways=[ [e2e-llm-inference-service] make_router_gateway( [e2e-llm-inference-service] "router-gateway-1", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="custom-route-timeout-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="router-with-refs-test", [e2e-llm-inference-service] expected_gateway="router-gateway-1", [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda tc: create_router_resources( [e2e-llm-inference-service] gateways=[ [e2e-llm-inference-service] make_router_gateway( [e2e-llm-inference-service] "router-gateway-1", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] routes=[ [e2e-llm-inference-service] make_router_main_route( [e2e-llm-inference-service] "router-route-1", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] "router-gateway-1", [e2e-llm-inference-service] "router-with-refs-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] make_router_health_route( [e2e-llm-inference-service] "router-route-2", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] "router-gateway-1", [e2e-llm-inference-service] "router-with-refs-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=["router-managed", "workload-pd-cpu", "model-fb-opt-125m"], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="custom-route-timeout-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="router-with-refs-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] expected_gateway="router-gateway-2", [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda tc: create_router_resources( [e2e-llm-inference-service] gateways=[ [e2e-llm-inference-service] make_router_gateway( [e2e-llm-inference-service] "router-gateway-2", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] routes=[ [e2e-llm-inference-service] make_router_main_route( [e2e-llm-inference-service] "router-route-3", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] "router-gateway-2", [e2e-llm-inference-service] "router-with-refs-pd-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] make_router_health_route( [e2e-llm-inference-service] "router-route-4", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] "router-gateway-2", [e2e-llm-inference-service] "router-with-refs-pd-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-dp-ep-gpu", [e2e-llm-inference-service] "workload-dp-ep-prefill-gpu", [e2e-llm-inference-service] "model-deepseek-v2-lite", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="Delve into the multifaceted implications of a fully disaggregated cloud architecture, specifically " [e2e-llm-inference-service] "where the compute plane (P) and the data plane (D) are independently deployed and managed for a " [e2e-llm-inference-service] "geographically distributed, high-throughput, low-latency microservices ecosystem. Beyond the " [e2e-llm-inference-service] "fundamental challenges of network latency and data consistency, elaborate on the advanced " [e2e-llm-inference-service] "considerations and trade-offs inherent in such a setup: 1. Network Architecture and Protocols: " [e2e-llm-inference-service] "How would the network fabric and underlying protocols (e.g., RDMA, custom transport layers) need to " [e2e-llm-inference-service] "evolve to support optimal performance and minimize inter-plane communication overhead, especially for " [e2e-llm-inference-service] "synchronous operations? Discuss the role of network programmability (e.g., SDN, P4) in dynamically " [e2e-llm-inference-service] "optimizing routing and traffic flow between P and D. 2. Advanced Data Consistency and Durability: " [e2e-llm-inference-service] "Explore sophisticated data consistency models (e.g., causal consistency, strong eventual consistency) " [e2e-llm-inference-service] "and their applicability in balancing performance and data integrity across a globally distributed data plane. " [e2e-llm-inference-service] "Detail strategies for ensuring data durability and fault tolerance, including multi-region replication, " [e2e-llm-inference-service] "intelligent partitioning, and recovery mechanisms in the event of partial or full plane failures. " [e2e-llm-inference-service] "3. Dynamic Resource Orchestration and Cost Optimization: Analyze how an orchestration layer would intelligently " [e2e-llm-inference-service] "manage the independent scaling of compute (P) and data (D) resources, considering fluctuating workloads, " [e2e-llm-inference-service] "cost efficiency, and performance targets (e.g., using predictive analytics for resource provisioning). " [e2e-llm-inference-service] "Discuss mechanisms for dynamically reallocating compute nodes to different data partitions based on " [e2e-llm-inference-service] "workload patterns and data locality, potentially involving live migration strategies. " [e2e-llm-inference-service] "4. Security and Compliance in a Distributed Landscape: Address the enhanced security perimeter " [e2e-llm-inference-service] "challenges, including securing communication channels between P and D (encryption in transit, mutual TLS), " [e2e-llm-inference-service] "fine-grained access control to data at rest and in motion, and identity management across disaggregated " [e2e-llm-inference-service] "components. Discuss how such an architecture impacts compliance with regulatory frameworks (e.g., GDPR, HIPAA) " [e2e-llm-inference-service] "concerning data sovereignty, privacy, and auditability. 5. Operational Complexity and Observability: " [e2e-llm-inference-service] "Examine the increased complexity in monitoring, logging, and tracing across highly decoupled compute and " [e2e-llm-inference-service] "data planes. What specialized tooling and practices (e.g., distributed tracing with OpenTelemetry, advanced AIOps) " [e2e-llm-inference-service] "would be essential? How would incident response and troubleshooting differ in this disaggregated environment " [e2e-llm-inference-service] "compared to traditional integrated systems? Consider the challenges of pinpointing root causes across " [e2e-llm-inference-service] "independent failures. 6. Real-world Applicability and Future Trends: Identify specific industries " [e2e-llm-inference-service] "or use cases (e.g., high-frequency trading, IoT edge processing, large language model inference) " [e2e-llm-inference-service] "where the benefits of P/D disaggregation would strongly outweigh its complexities. " [e2e-llm-inference-service] "Conclude by speculating on emerging technologies or paradigms (e.g., serverless compute functions " [e2e-llm-inference-service] "directly interacting with object storage, in-memory disaggregation) that could further drive or " [e2e-llm-inference-service] "transform P/D disaggregation in cloud computing.", [e2e-llm-inference-service] max_tokens=2000, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_gpu, [e2e-llm-inference-service] pytest.mark.cluster_nvidia, [e2e-llm-inference-service] pytest.mark.cluster_nvidia_roce, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-no-scheduler", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.no_scheduler, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="This test simulates DP+EP that can run on CPU, the idea is to test the LWS-based deployment, " [e2e-llm-inference-service] "but without the resources requirements for DP+EP (GPUs and ROCe/IB).", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_multi_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler config tests [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-inline-config-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Chat completions endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] model_name="Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-configmap-ref", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-configmap-ref-test", [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda tc: create_scheduler_configmap(namespace=tc.namespace) [e2e-llm-inference-service] ], [e2e-llm-inference-service] after_test=[ [e2e-llm-inference-service] lambda tc: delete_scheduler_configmap(namespace=tc.namespace) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-replicas", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-ha-replicas-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-custom-template", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-custom-template-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler v0.6 → v0.7 migration tests. [e2e-llm-inference-service] # Deploy v0.6-style configs and verify the controller migrates them [e2e-llm-inference-service] # so the v0.7 scheduler boots successfully. [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-v06-pd-config-migration", [e2e-llm-inference-service] "workload-llmd-simulator-pd", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-v06-pd-migration-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-v06-nonzero-threshold-migration", [e2e-llm-inference-service] "workload-llmd-simulator-pd", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-v06-threshold-migration-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Standalone tokenizer — clean path: token-producer in inline config [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-tokenizer-kvcache", [e2e-llm-inference-service] "workload-llmd-simulator-kvcache", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="tokenizer-clean-path-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Standalone tokenizer — migration path: legacy precise-prefix-cache-scorer [e2e-llm-inference-service] # triggers auto-provisioned tokenizer without explicit tokenizer:{} field [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-precise-prefix-cache-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator-kvcache", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="tokenizer-migration-path-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Models endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="data"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/chat/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — LoRA adapter [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] model_name="publishers/{namespace}/models/lora-adapter-1", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "publishers/{namespace}/models/lora-adapter-1" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/lora-adapter-1", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/models (base + LoRA) [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=assert_models_contains( [e2e-llm-inference-service] "facebook/opt-125m", [e2e-llm-inference-service] "publishers/{namespace}/models/facebook/opt-125m", [e2e-llm-inference-service] "lora-adapter-1", [e2e-llm-inference-service] "publishers/{namespace}/models/lora-adapter-1", [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # PVC storage tests -- validate direct PVC volume mount with real vLLM serving [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] before_test=[lambda tc: ensure_pvc_with_model(namespace=tc.namespace)], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] before_test=[lambda tc: ensure_pvc_with_model(namespace=tc.namespace)], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] before_test=[lambda tc: ensure_pvc_with_model(namespace=tc.namespace)], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_multi_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_inference_service(test_case: TestCase): # noqa: F811 [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = KServeClient( [e2e-llm-inference-service] config_file=os.environ.get("KUBECONFIG", "~/.kube/config"), [e2e-llm-inference-service] client_configuration=client.Configuration(), [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] prefix = test_case.log_prefix [e2e-llm-inference-service] [e2e-llm-inference-service] test_failed = False [e2e-llm-inference-service] try: [e2e-llm-inference-service] print(f"{prefix} Creating LLMInferenceService {service_name}") [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] print(f"{prefix} Waiting for LLMInferenceService {service_name} to be ready") [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:866: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...pu-ll-699c687c'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-cfc8d654'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-07-28T11:10:30.224608', start_time = 1785237030.2248735 [e2e-llm-inference-service] duration = 900.7000124454498, timestamp_end = '2026-07-28T11:25:30.924890' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....p-ep-cpu-ll-699c687c'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-cfc8d654'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1376: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7f7bf842d800> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1387: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T11:11:16Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T11:11:01Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T11:10:49Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T11:11:34Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T11:11:34Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T11:11:34Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T11:10:49Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T11:10:49Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1371: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:82 Created test namespace e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret seaweedfs-s3-creds from kserve-ci-e2e-test to e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret storage-config from kserve-ci-e2e-test to e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:178 Patched default SA in e2e-test-llm-inference-service-7c3633d1 with secret seaweedfs-s3-creds [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:162 ConfigMap odh-kserve-custom-ca-bundle already exists in e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:159 Copied ConfigMap odh-trusted-ca-bundle from kserve-ci-e2e-test to e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1959 Created PVC e2e-pvc-model-storage in namespace e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2099 Created model download Job e2e-pvc-model-download in namespace e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2124 Model download Job e2e-pvc-model-download completed successfully [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig router-managed-llmisvc-model-pv-2f105a64 in namespace e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig router-managed-llmisvc-model-pv-2f105a64 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig router-managed-llmisvc-model-pv-2f105a64 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig workload-simulated-dp-ep-cpu-ll-699c687c in namespace e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig workload-simulated-dp-ep-cpu-ll-699c687c [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig workload-simulated-dp-ep-cpu-ll-699c687c [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig model-pvc-llmisvc-model-pvc-rou-cfc8d654 in namespace e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig model-pvc-llmisvc-model-pvc-rou-cfc8d654 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig model-pvc-llmisvc-model-pvc-rou-cfc8d654 [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_inference_service] [2026-07-28T11:10:29.672375] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-simulated-dp-ep-cpu', 'model-pvc'], prompt='KServe is a', service_name='llmisvc-model-pvc-router-manage-2577e794', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, namespace='e2e-test-llm-inference-service-7c3633d1', before_test=[ at 0x7f7bf97ff420>], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-pvc-router-manage-2577e794', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-inference-service-7c3633d1', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-pv-2f105a64'}, [e2e-llm-inference-service] {'name': 'workload-simulated-dp-ep-cpu-ll-699c687c'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-cfc8d654'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-28T11:10:29.684655] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-pvc-router-manage-2577e794', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-inference-service-7c3633d1', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-pv-2f105a64'}, [e2e-llm-inference-service] {'name': 'workload-simulated-dp-ep-cpu-ll-699c687c'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-cfc8d654'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-28T11:10:30.224422] end - ✅ in 0.540s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-28T11:10:30.224608] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-pvc-router-manage-2577e794', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-inference-service-7c3633d1', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-pv-2f105a64'}, [e2e-llm-inference-service] {'name': 'workload-simulated-dp-ep-cpu-ll-699c687c'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-cfc8d654'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'RouterReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T11:10:49Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T11:10:49Z', 'message': 'Inference Pool e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T11:10:49Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T11:10:49Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T11:10:49Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T11:10:49Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T11:10:49Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T11:10:49Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'RouterReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T11:11:01Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-route: "False" (reason "BackendNotFound", message "backend(llmisvc-model-pvc-router-manage-2577e794-inference--ip-950f33a3.e2e-test-llm-inference-service-7c3633d1.svc.cluster.local) not found")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T11:11:01Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T11:10:49Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T11:11:01Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-route: "False" (reason "BackendNotFound", message "backend(llmisvc-model-pvc-router-manage-2577e794-inference--ip-950f33a3.e2e-test-llm-inference-service-7c3633d1.svc.cluster.local) not found")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T11:11:01Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-route: "False" (reason "BackendNotFound", message "backend(llmisvc-model-pvc-router-manage-2577e794-inference--ip-950f33a3.e2e-test-llm-inference-service-7c3633d1.svc.cluster.local) not found")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T11:11:01Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T11:10:49Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T11:10:49Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'RouterReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T11:11:16Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T11:11:01Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T11:10:49Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T11:11:16Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T11:11:16Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T11:11:01Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T11:10:49Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T11:10:49Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T11:11:16Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T11:11:01Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T11:10:49Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T11:11:34Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T11:11:34Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T11:11:34Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T11:10:49Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T11:10:49Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1391 Timed out waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T11:11:16Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T11:11:01Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T11:10:49Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T11:11:34Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T11:11:34Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T11:11:34Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T11:10:49Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T11:10:49Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-07-28T11:25:30.924890] end - ❌ 900.700s: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T11:11:16Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T11:11:01Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T11:10:49Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T11:11:34Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T11:11:34Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T11:11:34Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T11:10:49Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T11:10:49Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:903 [router-managed-workload-simulated-dp-ep-cpu-model-pvc] ❌ ERROR: Failed to call llm inference service llmisvc-model-pvc-router-manage-2577e794: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T11:11:16Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T11:11:01Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T11:10:49Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T11:11:34Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T11:11:34Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T11:11:34Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T11:10:49Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T11:10:49Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:243 # Diagnostics for 'llmisvc-model-pvc-router-manage-2577e794' in 'e2e-test-llm-inference-service-7c3633d1' [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:244 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:247 # LLMInferenceService llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:256 apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] security.opendatahub.io/enable-auth: 'false' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T11:10:29Z' [e2e-llm-inference-service] finalizers: [e2e-llm-inference-service] - serving.kserve.io/llmisvc-finalizer [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:security.opendatahub.io/enable-auth: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:baseRefs: {} [e2e-llm-inference-service] manager: OpenAPI-Generator [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T11:10:29Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:finalizers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] v:"serving.kserve.io/llmisvc-finalizer": {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T11:10:34Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:addresses: {} [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-router-route: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-scheduler: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-scheduler-latency-predictor: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-tokenizer: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-tracing: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-worker-data-parallel: {} [e2e-llm-inference-service] f:appliedConfigs: {} [e2e-llm-inference-service] f:conditions: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:router: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:gateways: {} [e2e-llm-inference-service] f:scheduler: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:inferencePool: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:service: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:url: {} [e2e-llm-inference-service] f:workloads: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:primary: {} [e2e-llm-inference-service] f:scheduler: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-28T11:11:34Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] resourceVersion: '69185' [e2e-llm-inference-service] uid: b4db031d-a9f8-4c49-a6d1-98105ac4d3e2 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] baseRefs: [e2e-llm-inference-service] - name: router-managed-llmisvc-model-pv-2f105a64 [e2e-llm-inference-service] - name: workload-simulated-dp-ep-cpu-ll-699c687c [e2e-llm-inference-service] - name: model-pvc-llmisvc-model-pvc-rou-cfc8d654 [e2e-llm-inference-service] model: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uri: '' [e2e-llm-inference-service] status: [e2e-llm-inference-service] addresses: [e2e-llm-inference-service] - name: gateway-external-model-routing [e2e-llm-inference-service] url: http://ac8a2e4f596c64362acf1a34ef0ad9a3-2038940287.us-east-1.elb.amazonaws.com/ [e2e-llm-inference-service] - name: gateway-external [e2e-llm-inference-service] url: http://ac8a2e4f596c64362acf1a34ef0ad9a3-2038940287.us-east-1.elb.amazonaws.com/e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] - name: gateway-external [e2e-llm-inference-service] url: http://ac8a2e4f596c64362acf1a34ef0ad9a3-2038940287.us-east-1.elb.amazonaws.com/publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m [e2e-llm-inference-service] - name: gateway-internal-model-routing [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/ [e2e-llm-inference-service] - name: gateway-internal [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] - name: gateway-internal [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-template: kserve-config-llm-decode-template [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-worker-data-parallel: kserve-config-llm-decode-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-template: kserve-config-llm-prefill-template [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-worker-data-parallel: kserve-config-llm-prefill-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-router-route: kserve-config-llm-router-route [e2e-llm-inference-service] serving.kserve.io/config-llm-scheduler: kserve-config-llm-scheduler [e2e-llm-inference-service] serving.kserve.io/config-llm-scheduler-latency-predictor: kserve-config-llm-scheduler-latency-predictor [e2e-llm-inference-service] serving.kserve.io/config-llm-template: kserve-config-llm-template [e2e-llm-inference-service] serving.kserve.io/config-llm-tokenizer: kserve-config-llm-tokenizer [e2e-llm-inference-service] serving.kserve.io/config-llm-tracing: kserve-config-llm-tracing [e2e-llm-inference-service] serving.kserve.io/config-llm-worker-data-parallel: kserve-config-llm-worker-data-parallel [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T11:11:16Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: HTTPRoutesReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T11:11:01Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: InferencePoolReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T11:10:49Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: PresetsCombined [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T11:11:34Z' [e2e-llm-inference-service] message: LWS is progressing [e2e-llm-inference-service] reason: Progressing [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: Ready [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T11:11:34Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: RouterReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T11:11:34Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: SchedulerWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T11:10:49Z' [e2e-llm-inference-service] message: LWS is progressing [e2e-llm-inference-service] reason: Progressing [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: WorkerWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T11:10:49Z' [e2e-llm-inference-service] message: LWS is progressing [e2e-llm-inference-service] reason: Progressing [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: WorkloadsReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] url: http://ac8a2e4f596c64362acf1a34ef0ad9a3-2038940287.us-east-1.elb.amazonaws.com/e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:68 TIME NAMESPACE SOURCE TYPE REASON MESSAGE [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:69 -------------------------------------------------------------------------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None e2e-test-llm-inference-service-7c3633d1 Normal Scheduled Successfully assigned e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-vww8z to ip-10-0-141-50.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 11:10:16 e2e-test-llm-inference-service-7c3633d1 attachdetach-controller Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-22e6f62e-463f-4729-96e2-d0f264811ec3" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 11:10:16 e2e-test-llm-inference-service-7c3633d1 multus Normal AddedInterface Add eth0 [10.134.0.66/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 11:10:16 e2e-test-llm-inference-service-7c3633d1 kubelet/ip-10-0-141-50.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:527f9c4ef06f4e1d255c6c4b87333be01c67680fe0b3417b8fb7aeb5d5b61bbf" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 11:10:16 e2e-test-llm-inference-service-7c3633d1 kubelet/ip-10-0-141-50.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 11:10:16 e2e-test-llm-inference-service-7c3633d1 kubelet/ip-10-0-141-50.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 11:10:08 e2e-test-llm-inference-service-7c3633d1 job-controller Normal SuccessfulCreate Created pod: e2e-pvc-model-download-vww8z [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 11:10:26 e2e-test-llm-inference-service-7c3633d1 job-controller Normal Completed Job completed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 11:10:08 e2e-test-llm-inference-service-7c3633d1 persistentvolume-controller Normal WaitForFirstConsumer waiting for first consumer to be created before binding [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 11:10:08 e2e-test-llm-inference-service-7c3633d1 ebs.csi.aws.com_aws-ebs-csi-driver-controller-d96bdd68d-rh6ng_afb65174-87af-46ee-b050-57713714920e Normal Provisioning External provisioner is provisioning volume for claim "e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-storage" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 11:10:08 e2e-test-llm-inference-service-7c3633d1 persistentvolume-controller Normal ExternalProvisioning Waiting for a volume to be created either by the external provisioner 'ebs.csi.aws.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 11:10:10 e2e-test-llm-inference-service-7c3633d1 ebs.csi.aws.com_aws-ebs-csi-driver-controller-d96bdd68d-rh6ng_afb65174-87af-46ee-b050-57713714920e Normal ProvisioningSucceeded Successfully provisioned volume pvc-22e6f62e-463f-4729-96e2-d0f264811ec3 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None e2e-test-llm-inference-service-7c3633d1 Normal Scheduled Successfully assigned e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1 to ip-10-0-131-88.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 11:10:50 e2e-test-llm-inference-service-7c3633d1 attachdetach-controller Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-22e6f62e-463f-4729-96e2-d0f264811ec3" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 11:10:54 e2e-test-llm-inference-service-7c3633d1 multus Normal AddedInterface Add eth0 [10.133.0.65/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 11:10:54 e2e-test-llm-inference-service-7c3633d1 kubelet/ip-10-0-131-88.ec2.internal Normal Pulled Container image "vllm/vllm-openai-cpu:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 11:10:54 e2e-test-llm-inference-service-7c3633d1 kubelet/ip-10-0-131-88.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 11:10:54 e2e-test-llm-inference-service-7c3633d1 kubelet/ip-10-0-131-88.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None e2e-test-llm-inference-service-7c3633d1 Normal Scheduled Successfully assigned e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 to ip-10-0-130-235.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 11:10:48 e2e-test-llm-inference-service-7c3633d1 statefulset-controller Normal SuccessfulCreate create Pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1 in StatefulSet llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 successful [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 11:10:50 e2e-test-llm-inference-service-7c3633d1 attachdetach-controller Warning FailedAttachVolume Multi-Attach error for volume "pvc-22e6f62e-463f-4729-96e2-d0f264811ec3" Volume is already used by pod(s) llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 11:10:48 e2e-test-llm-inference-service-7c3633d1 leaderworkerset Normal CreatingRevision Creating revision with key 7bc46f896d for a newly created LeaderWorkerSet [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 11:10:48 e2e-test-llm-inference-service-7c3633d1 leaderworkerset Normal GroupsProgressing Created leader statefulset llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 11:10:48 e2e-test-llm-inference-service-7c3633d1 leaderworkerset Normal GroupsProgressing Replicas are progressing, with 0 groups ready of total 1 groups [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 11:10:48 e2e-test-llm-inference-service-7c3633d1 statefulset-controller Normal SuccessfulCreate create Pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 in StatefulSet llmisvc-model-pvc-router-manage-2577e794-kserve-mn successful [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 11:10:48 e2e-test-llm-inference-service-7c3633d1 leaderworkerset Normal GroupsProgressing Created worker statefulset for leader pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 11:10:45 e2e-test-llm-inference-service-7c3633d1 OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-pvc-router-manage-2577e794-kserve-route-authn: failed to get HTTPRoute e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-pvc-router-manage-2577e794-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 11:10:48 e2e-test-llm-inference-service-7c3633d1 LLMInferenceServiceController Normal Created Created v1.Secret e2e-test-llm-inference-service-7c3633d1/llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 11:10:48 e2e-test-llm-inference-service-7c3633d1 LLMInferenceServiceController Normal Created Created v1.RoleBinding e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-scc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 11:10:48 e2e-test-llm-inference-service-7c3633d1 LLMInferenceServiceController Normal Created Created v1.ServiceAccount e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 11:10:48 e2e-test-llm-inference-service-7c3633d1 LLMInferenceServiceController Normal Created Created v1.LeaderWorkerSet e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 11:10:48 e2e-test-llm-inference-service-7c3633d1 LLMInferenceServiceController Normal Created Created v1.Service e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 11:10:48 e2e-test-llm-inference-service-7c3633d1 LLMInferenceServiceController Normal Created Created v1.ServiceAccount e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 11:10:48 e2e-test-llm-inference-service-7c3633d1 LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /e2e-test-llm-inference-service-5571cf2b7477d60b74b195d7a6a8f03e [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 11:10:48 e2e-test-llm-inference-service-7c3633d1 LLMInferenceServiceController Normal Created Created v1.Role e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 11:10:48 e2e-test-llm-inference-service-7c3633d1 LLMInferenceServiceController Normal Created Created v1.RoleBinding e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 11:11:16 e2e-test-llm-inference-service-7c3633d1 LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 11:20:49 e2e-test-llm-inference-service-7c3633d1 LLMInferenceServiceController Normal Updated Updated v1.Secret e2e-test-llm-inference-service-7c3633d1/llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 11:11:00 e2e-test-llm-inference-service-7c3633d1 LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 11:10:48 e2e-test-llm-inference-service-7c3633d1 replicaset-controller Normal SuccessfulCreate Created pod: llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schekxjwb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 11:10:48 e2e-test-llm-inference-service-7c3633d1 deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheduler-66dcb4d65d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None e2e-test-llm-inference-service-7c3633d1 Normal Scheduled Successfully assigned e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schekxjwb to ip-10-0-141-50.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 11:10:49 e2e-test-llm-inference-service-7c3633d1 multus Normal AddedInterface Add eth0 [10.134.0.67/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 11:10:49 e2e-test-llm-inference-service-7c3633d1 kubelet/ip-10-0-141-50.ec2.internal Normal Pulled Container image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 11:10:49 e2e-test-llm-inference-service-7c3633d1 kubelet/ip-10-0-141-50.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 11:10:49 e2e-test-llm-inference-service-7c3633d1 kubelet/ip-10-0-141-50.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:172 ### Pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 (phase=Pending) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:212 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:228 # -- logs (current): unavailable ((400) [e2e-llm-inference-service] Reason: Bad Request [e2e-llm-inference-service] HTTP response headers: HTTPHeaderDict({'Audit-Id': '39472ce4-150c-4606-9684-1da326b8566a', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'Date': 'Tue, 28 Jul 2026 11:25:31 GMT', 'Content-Length': '236'}) [e2e-llm-inference-service] HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"container \"main\" in pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" is waiting to start: ContainerCreating","reason":"BadRequest","code":400} [e2e-llm-inference-service] [e2e-llm-inference-service] ) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:172 ### Pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1 (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:212 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:225 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:226 INFO 07-28 11:11:06 [importing.py:44] Triton is installed but 0 active driver(s) found (expected 1). Disabling Triton to prevent runtime errors. [e2e-llm-inference-service] INFO 07-28 11:11:06 [importing.py:68] Triton not installed or not compatible; certain GPU-related functions will not be available. [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 11:11:12 [utils.py:299] [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 11:11:12 [utils.py:299] █ █ █▄ ▄█ [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 11:11:12 [utils.py:299] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.19.0 [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 11:11:12 [utils.py:299] █▄█▀ █ █ █ █ model /mnt/models [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 11:11:12 [utils.py:299] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀ [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 11:11:12 [utils.py:299] [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 11:11:12 [utils.py:233] non-default args: {'model_tag': '/mnt/models', 'ssl_keyfile': '/var/run/kserve/tls/tls.key', 'ssl_certfile': '/var/run/kserve/tls/tls.crt', 'enable_ssl_refresh': True, 'model': '/mnt/models', 'served_model_name': ['facebook/opt-125m']} [e2e-llm-inference-service] (APIServer pid=1) WARNING 07-28 11:11:12 [arg_utils.py:1390] The global random seed is set to 0. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM. [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 11:11:29 [model.py:549] Resolved architecture: OPTForCausalLM [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 11:11:29 [model.py:1678] Using max model len 2048 [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 11:11:29 [vllm.py:790] Asynchronous scheduling is enabled. [e2e-llm-inference-service] INFO 07-28 11:11:42 [importing.py:44] Triton is installed but 0 active driver(s) found (expected 1). Disabling Triton to prevent runtime errors. [e2e-llm-inference-service] INFO 07-28 11:11:42 [importing.py:68] Triton not installed or not compatible; certain GPU-related functions will not be available. [e2e-llm-inference-service] (EngineCore pid=41) INFO 07-28 11:11:47 [core.py:105] Initializing a V1 LLM engine (v0.19.0) with config: model='/mnt/models', speculative_config=None, tokenizer='/mnt/models', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=2048, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cpu, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=facebook/opt-125m, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': , 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_images_per_batch': 0, 'compile_sizes': None, 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'dce': True, 'nan_asserts': False, 'epilogue_fusion': True, 'cpp.dynamic_threads': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': None, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} [e2e-llm-inference-service] (EngineCore pid=41) INFO 07-28 11:11:48 [cpu_worker.py:236] auto thread-binding list (id, physical core): [(4, 0), (5, 1), (6, 2), (7, 3)] [e2e-llm-inference-service] [W728 11:11:48.228533376 utils.cpp:76] Warning: numa_migrate_pages failed. errno: 1 (function init_cpu_threads_env) [e2e-llm-inference-service] [W728 11:11:48.228558120 utils.cpp:103] Warning: NUMA binding: Using MEMBIND policy for memory allocation on the NUMA nodes (0). Memory allocations will be strictly bound to these NUMA nodes. (function init_cpu_threads_env) [e2e-llm-inference-service] (EngineCore pid=41) INFO 07-28 11:11:48 [cpu_worker.py:109] OMP threads binding of Process 41: [e2e-llm-inference-service] (EngineCore pid=41) INFO 07-28 11:11:48 [cpu_worker.py:109] OMP tid: 41, core 4 [e2e-llm-inference-service] (EngineCore pid=41) INFO 07-28 11:11:48 [cpu_worker.py:109] OMP tid: 58, core 5 [e2e-llm-inference-service] (EngineCore pid=41) INFO 07-28 11:11:48 [cpu_worker.py:109] OMP tid: 59, core 6 [e2e-llm-inference-service] (EngineCore pid=41) INFO 07-28 11:11:48 [cpu_worker.py:109] OMP tid: 60, core 7 [e2e-llm-inference-service] (EngineCore pid=41) INFO 07-28 11:11:48 [cpu_worker.py:109] [e2e-llm-inference-service] (EngineCore pid=41) INFO 07-28 11:11:48 [parallel_state.py:1400] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.133.0.65:33845 backend=gloo [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] (EngineCore pid=41) INFO 07-28 11:11:48 [parallel_state.py:1716] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A [e2e-llm-inference-service] (EngineCore pid=41) INFO 07-28 11:11:48 [cpu_model_runner.py:71] Starting to load model /mnt/models... [e2e-llm-inference-service] (EngineCore pid=41) Loading pt checkpoint shards: 0% Completed | 0/1 [00:00}, FlowControl: {MaxBytes: unlimited, MaxRequests: unlimited, SaturationDetector: {PluginRef: utilization-detector}}, RequestHandler: {Parsers: [{PluginRef: openai-parser}, {PluginRef: anthropic-parser}, {PluginRef: vllmhttp-parser}]}}"} [e2e-llm-inference-service] {"level":"info","ts":1785237049.1641352,"caller":"approximateprefix/plugin.go:88","msg":"Prefix DataProducer initialized","config":{"autoTune":true,"blockSizeTokens":16,"blockSize":0,"maxPrefixBlocksToMatch":2048,"maxPrefixTokensToMatch":131072,"lruCapacityPerServer":31250}} [e2e-llm-inference-service] {"level":"info","ts":1785237049.1641974,"caller":"approximateprefix/plugin.go:111","msg":"WARNING: configured blockSizeTokens is below the recommended minimum, overriding it.","blockSizeTokens":16,"minimum":64,"issue":"https://github.com/llm-d/llm-d-router/issues/1158"} [e2e-llm-inference-service] {"level":"info","ts":1785237049.1642148,"caller":"datalayer/data_graph.go:116","msg":"auto-created default producer","producer":"approx-prefix-cache-producer/approx-prefix-cache-producer","dataKey":"PrefixCacheMatchInfoDataKey/approx-prefix-cache-producer","consumer":"no-hit-lru-scorer"} [e2e-llm-inference-service] {"level":"info","ts":1785237049.1642406,"caller":"datalayer/data_graph.go:116","msg":"auto-created default producer","producer":"token-producer/token-producer","dataKey":"TokenizedPrompt/token-producer","consumer":"approx-prefix-cache-producer"} [e2e-llm-inference-service] {"level":"info","ts":1785237049.164331,"caller":"runner/runner.go:685","msg":"loaded configuration from file/text successfully"} [e2e-llm-inference-service] {"level":"info","ts":1785237049.1643405,"logger":"setup","caller":"runner/runner.go:308","msg":"EPP config after phase two","config":"{SchedulerConfig:{ProfileHandler: single-profile-handler/single-profile-handler, Profiles: map[default:{Filters: [], Scorers: [queue-scorer/queue-scorer: 2.000000, kv-cache-utilization-scorer/kv-cache-utilization-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000, no-hit-lru-scorer/no-hit-lru-scorer: 2.000000], Picker: max-score-picker/max-score-picker}]} SaturationDetector:0xc00069cac0 DataConfig:{Sources:[{Plugin:0xc0003127e0 Extractors:[0xc00069cd40]}]} FlowControlConfig: ParserRegistry:0xc00069d500}"} [e2e-llm-inference-service] {"level":"info","ts":1785237049.1827302,"logger":"setup","caller":"runner/runner.go:352","msg":"Setting pprof handlers"} [e2e-llm-inference-service] {"level":"info","ts":1785237049.1827736,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/mutex"} [e2e-llm-inference-service] {"level":"info","ts":1785237049.1827884,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/"} [e2e-llm-inference-service] {"level":"info","ts":1785237049.1827931,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/symbol"} [e2e-llm-inference-service] {"level":"info","ts":1785237049.1827981,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/heap"} [e2e-llm-inference-service] {"level":"info","ts":1785237049.1828024,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/cmdline"} [e2e-llm-inference-service] {"level":"info","ts":1785237049.1828065,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/profile"} [e2e-llm-inference-service] {"level":"info","ts":1785237049.182811,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/trace"} [e2e-llm-inference-service] {"level":"info","ts":1785237049.1828153,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/goroutine"} [e2e-llm-inference-service] {"level":"info","ts":1785237049.182823,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/allocs"} [e2e-llm-inference-service] {"level":"info","ts":1785237049.1828275,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/threadcreate"} [e2e-llm-inference-service] {"level":"info","ts":1785237049.1828313,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/block"} [e2e-llm-inference-service] {"level":"info","ts":1785237049.1828382,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/plugins/state"} [e2e-llm-inference-service] {"level":"info","ts":1785237049.182843,"logger":"setup","caller":"runner/runner.go:373","msg":"parsed config","scheduler-config":"{ProfileHandler: single-profile-handler/single-profile-handler, Profiles: map[default:{Filters: [], Scorers: [queue-scorer/queue-scorer: 2.000000, kv-cache-utilization-scorer/kv-cache-utilization-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000, no-hit-lru-scorer/no-hit-lru-scorer: 2.000000], Picker: max-score-picker/max-score-picker}]}"} [e2e-llm-inference-service] {"level":"info","ts":1785237049.1828754,"logger":"setup","caller":"datalayer/runtime.go:99","msg":"Configuring datalayer runtime","numSources":1} [e2e-llm-inference-service] {"level":"info","ts":1785237049.1828847,"logger":"setup","caller":"datalayer/runtime.go:118","msg":"Processing source","source":"metrics-data-source","numExtractors":1} [e2e-llm-inference-service] {"level":"info","ts":1785237049.182899,"logger":"setup","caller":"datalayer/runtime.go:147","msg":"Source configured","source":"metrics-data-source","extractors":["core-metrics-extractor/core-metrics-extractor"]} [e2e-llm-inference-service] {"level":"info","ts":1785237049.1829112,"logger":"setup","caller":"datalayer/runtime.go:206","msg":"Datalayer runtime configured","pollers":1,"notifiers":0,"endpointSources":0} [e2e-llm-inference-service] {"level":"info","ts":1785237049.1829236,"logger":"setup","caller":"runner/runner.go:833","msg":"Experimental Flow Control layer is disabled, using legacy admission control"} [e2e-llm-inference-service] {"level":"info","ts":1785237049.183011,"logger":"setup","caller":"runner/runner.go:721","msg":"ExtProc server runner added to manager."} [e2e-llm-inference-service] {"level":"info","ts":1785237049.1830287,"logger":"setup","caller":"runner/runner.go:260","msg":"Controller manager starting"} [e2e-llm-inference-service] {"level":"info","ts":1785237049.183059,"logger":"controller-runtime.metrics","caller":"server/server.go:208","msg":"Starting metrics server"} [e2e-llm-inference-service] {"level":"info","ts":1785237049.1832635,"caller":"runnable/grpc.go:35","msg":"gRPC server starting","name":"health"} [e2e-llm-inference-service] {"level":"info","ts":1785237049.183638,"caller":"runnable/grpc.go:43","msg":"gRPC server listening","name":"health","port":9003} [e2e-llm-inference-service] {"level":"info","ts":1785237049.183706,"logger":"controller-runtime.metrics","caller":"server/server.go:247","msg":"Serving metrics server","bindAddress":":9090","secure":false} [e2e-llm-inference-service] {"level":"info","ts":1785237049.1838546,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","source":"kind source: *v1.InferencePool"} [e2e-llm-inference-service] {"level":"info","ts":1785237049.1838982,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferencemodelrewrite","controllerGroup":"llm-d.ai","controllerKind":"InferenceModelRewrite","source":"kind source: *v1alpha2.InferenceModelRewrite"} [e2e-llm-inference-service] {"level":"info","ts":1785237049.183965,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"pod","controllerGroup":"","controllerKind":"Pod","source":"kind source: *v1.Pod"} [e2e-llm-inference-service] {"level":"info","ts":1785237049.184125,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferenceobjective","controllerGroup":"llm-d.ai","controllerKind":"InferenceObjective","source":"kind source: *v1alpha2.InferenceObjective"} [e2e-llm-inference-service] {"level":"info","ts":1785237049.1845944,"caller":"runnable/grpc.go:35","msg":"gRPC server starting","name":"ext-proc"} [e2e-llm-inference-service] {"level":"info","ts":1785237049.184715,"caller":"runnable/grpc.go:43","msg":"gRPC server listening","name":"ext-proc","port":9002} [e2e-llm-inference-service] {"level":"info","ts":1785237049.1887734,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.InferenceObjective","reflector":"go/pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1785237049.1888182,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.InferenceModelRewrite","reflector":"go/pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1785237049.1890714,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.InferencePool","reflector":"go/pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1785237049.1891913,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Pod","reflector":"go/pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1785237049.2853343,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferenceobjective","controllerGroup":"llm-d.ai","controllerKind":"InferenceObjective"} [e2e-llm-inference-service] {"level":"info","ts":1785237049.2853425,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferencemodelrewrite","controllerGroup":"llm-d.ai","controllerKind":"InferenceModelRewrite"} [e2e-llm-inference-service] {"level":"info","ts":1785237049.2853665,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferenceobjective","controllerGroup":"llm-d.ai","controllerKind":"InferenceObjective","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1785237049.2853706,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferencemodelrewrite","controllerGroup":"llm-d.ai","controllerKind":"InferenceModelRewrite","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1785237049.2853932,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool"} [e2e-llm-inference-service] {"level":"info","ts":1785237049.2853982,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1785237049.2855163,"caller":"controller/inferencepool_reconciler.go:46","msg":"Reconciling InferencePool","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"llmisvc-model-pvc-router-manage-2577e794-inference-pool","namespace":"e2e-test-llm-inference-service-7c3633d1"},"namespace":"e2e-test-llm-inference-service-7c3633d1","name":"llmisvc-model-pvc-router-manage-2577e794-inference-pool","reconcileID":"6c5cab34-feab-45ed-a349-4a7b152794f7"} [e2e-llm-inference-service] {"level":"info","ts":1785237049.385468,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"pod","controllerGroup":"","controllerKind":"Pod"} [e2e-llm-inference-service] {"level":"info","ts":1785237049.3855026,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"pod","controllerGroup":"","controllerKind":"Pod","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1785237060.5396507,"caller":"controller/inferencepool_reconciler.go:46","msg":"Reconciling InferencePool","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"llmisvc-model-pvc-router-manage-2577e794-inference-pool","namespace":"e2e-test-llm-inference-service-7c3633d1"},"namespace":"e2e-test-llm-inference-service-7c3633d1","name":"llmisvc-model-pvc-router-manage-2577e794-inference-pool","reconcileID":"f6399798-62bb-4dab-84e4-34ce93e8a467"} [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-service [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] uid: 00f36ccc-efa4-4ecb-9dc6-6da4dd743b56 [e2e-llm-inference-service] resourceVersion: '69040' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-28T11:11:20Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T11:11:20Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - ip: 10.134.0.67 [e2e-llm-inference-service] nodeName: ip-10-0-141-50.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] name: llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schekxjwb [e2e-llm-inference-service] uid: be54b512-094b-4f12-94ae-858699835a9e [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] uid: d94e3600-c59e-455c-ac5a-0cddb7332503 [e2e-llm-inference-service] resourceVersion: '68421' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-2577e794-kserve-mn- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] uid: 8cc1f64c-273d-47e1-add7-cfef2f51d3b0 [e2e-llm-inference-service] resourceVersion: '68448' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-leader [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] apps.kubernetes.io/pod-index: '0' [e2e-llm-inference-service] controller-revision-hash: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-5cf8fb667d [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/group-index: '0' [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/group-key: ae449cef3b581b6dbab6ee0440d253929da5cb58 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/template-revision-hash: 7bc46f896d [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/worker-index: '0' [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] statefulset.kubernetes.io/pod-name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.132.0.40/23"],"mac_address":"0a:58:0a:84:00:28","gateway_ips":["10.132.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.132.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.132.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.132.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.132.0.1"}],"ip_address":"10.132.0.40/23","gateway_ip":"10.132.0.1","role":"primary"}}' [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/size: '2' [e2e-llm-inference-service] openshift.io/scc: openshift-ai-llminferenceservice-scc [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: serviceaccount [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: StatefulSet [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] uid: b2882710-1544-4efd-82d3-b54864075665 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-130-235 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/size: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:apps.kubernetes.io/pod-index: {} [e2e-llm-inference-service] f:controller-revision-hash: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/name: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/template-revision-hash: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/worker-index: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:statefulset.kubernetes.io/pod-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b2882710-1544-4efd-82d3-b54864075665"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:add: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:hostname: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:subdomain: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 8Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] - name: kube-api-access-whstl [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - vllm [e2e-llm-inference-service] - serve [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --served-model-name [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --enable-ssl-refresh [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: LWS_LEADER_ADDRESS [e2e-llm-inference-service] value: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0.llmisvc-model-pvc-router-manage-2577e794-kserve-mn.e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] - name: LWS_GROUP_SIZE [e2e-llm-inference-service] value: '2' [e2e-llm-inference-service] - name: LWS_WORKER_INDEX [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-whstl [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] add: [e2e-llm-inference-service] - IPC_LOCK [e2e-llm-inference-service] - SYS_RAWIO [e2e-llm-inference-service] - NET_RAW [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1001000000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] nodeName: ip-10-0-130-235.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c32,c4 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-lfhh2 [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-zzdrr [e2e-llm-inference-service] hostname: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 [e2e-llm-inference-service] subdomain: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Pending [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] reason: ContainersNotReady [e2e-llm-inference-service] message: 'containers with unready status: [main]' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] reason: ContainersNotReady [e2e-llm-inference-service] message: 'containers with unready status: [main]' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] hostIP: 10.0.130.235 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.130.235 [e2e-llm-inference-service] startTime: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] waiting: [e2e-llm-inference-service] reason: ContainerCreating [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: false [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] imageID: '' [e2e-llm-inference-service] started: false [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-whstl [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1 [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] uid: 306cf4bf-a7df-4f67-ab1b-4e7547006c4a [e2e-llm-inference-service] resourceVersion: '68716' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-worker [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] apps.kubernetes.io/pod-index: '1' [e2e-llm-inference-service] controller-revision-hash: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-74696b7c6f [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/group-index: '0' [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/group-key: ae449cef3b581b6dbab6ee0440d253929da5cb58 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/template-revision-hash: 7bc46f896d [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/worker-index: '1' [e2e-llm-inference-service] statefulset.kubernetes.io/pod-name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.133.0.65/23"],"mac_address":"0a:58:0a:85:00:41","gateway_ips":["10.133.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.133.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.133.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.133.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.133.0.1"}],"ip_address":"10.133.0.65/23","gateway_ip":"10.133.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.133.0.65\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:85:00:41\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/leader-name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/size: '2' [e2e-llm-inference-service] openshift.io/scc: openshift-ai-llminferenceservice-scc [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: serviceaccount [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: StatefulSet [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 [e2e-llm-inference-service] uid: 7fec6a81-d121-42f6-93fe-7a509a63285d [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-131-88 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/leader-name: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/size: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:apps.kubernetes.io/pod-index: {} [e2e-llm-inference-service] f:controller-revision-hash: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/group-index: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/group-key: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/name: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/template-revision-hash: {} [e2e-llm-inference-service] f:statefulset.kubernetes.io/pod-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7fec6a81-d121-42f6-93fe-7a509a63285d"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:add: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:hostname: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:subdomain: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T11:10:54Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.133.0.65"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T11:10:54Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 8Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] - name: kube-api-access-vwqtj [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - vllm [e2e-llm-inference-service] - serve [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --served-model-name [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --enable-ssl-refresh [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: LWS_LEADER_ADDRESS [e2e-llm-inference-service] value: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0.llmisvc-model-pvc-router-manage-2577e794-kserve-mn.e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] - name: LWS_GROUP_SIZE [e2e-llm-inference-service] value: '2' [e2e-llm-inference-service] - name: LWS_WORKER_INDEX [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-vwqtj [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] add: [e2e-llm-inference-service] - IPC_LOCK [e2e-llm-inference-service] - SYS_RAWIO [e2e-llm-inference-service] - NET_RAW [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1001000000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] nodeName: ip-10-0-131-88.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c32,c4 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-lfhh2 [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-zzdrr [e2e-llm-inference-service] hostname: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1 [e2e-llm-inference-service] subdomain: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T11:10:54Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T11:10:54Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T11:10:54Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] hostIP: 10.0.131.88 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.131.88 [e2e-llm-inference-service] podIP: 10.133.0.65 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.133.0.65 [e2e-llm-inference-service] startTime: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-07-28T11:10:54Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: docker.io/vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] imageID: docker.io/vllm/vllm-openai-cpu@sha256:a8257c201fd2f696c146615987be61579aa2052367b73c27bd3fb1ab8e90bd4b [e2e-llm-inference-service] containerID: cri-o://e1d6685281b99bbc2aba88fb68908918e6b820d89aa81fe9c64fd8c707d99210 [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-vwqtj [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1001000000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schekxjwb [e2e-llm-inference-service] generateName: llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheduler-66dcb4d65d- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] uid: be54b512-094b-4f12-94ae-858699835a9e [e2e-llm-inference-service] resourceVersion: '69039' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 66dcb4d65d [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.10.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.134.0.67/23"],"mac_address":"0a:58:0a:86:00:43","gateway_ips":["10.134.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.134.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.134.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.134.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.134.0.1"}],"ip_address":"10.134.0.67/23","gateway_ip":"10.134.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.134.0.67\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:86:00:43\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheduler-66dcb4d65d [e2e-llm-inference-service] uid: 940e498a-aa18-4c19-b0da-15e8cca91d03 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-141-50 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"940e498a-aa18-4c19-b0da-15e8cca91d03"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T11:10:49Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T11:11:20Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.134.0.67"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kube-api-access-xpgbx [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: llm-d.ai/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n- type:\ [e2e-llm-inference-service] \ single-profile-handler\n- type: queue-scorer\n- type: kv-cache-utilization-scorer\n\ [e2e-llm-inference-service] - type: prefix-cache-scorer\n- type: no-hit-lru-scorer\n- type: max-score-picker\n\ [e2e-llm-inference-service] - parameters:\n scheme: https\n type: metrics-data-source\nschedulingProfiles:\n\ [e2e-llm-inference-service] - name: default\n plugins:\n - pluginRef: queue-scorer\n weight: 2\n -\ [e2e-llm-inference-service] \ pluginRef: kv-cache-utilization-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n\ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: no-hit-lru-scorer\n weight: 2\n - pluginRef:\ [e2e-llm-inference-service] \ max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kube-api-access-xpgbx [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1001000000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] nodeName: ip-10-0-141-50.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c32,c4 [e2e-llm-inference-service] fsGroup: 1001000000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-lfhh2 [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-2577e794-epp-sa-dockercfg-ktw7b [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T11:10:49Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T11:11:20Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T11:11:20Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] hostIP: 10.0.141.50 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.141.50 [e2e-llm-inference-service] podIP: 10.134.0.67 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.134.0.67 [e2e-llm-inference-service] startTime: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-07-28T11:10:49Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-llm-d-router-endpoint-picker@sha256:7fd1d61cb5505e026eebab0b9653758779c3dee241e37bea92a0bab961a48c50 [e2e-llm-inference-service] containerID: cri-o://6aa883e93344f3d3ccc0ab7363ed32aad66175d0f8f5515da82e434c1db123fc [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-xpgbx [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1001000000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1001000000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] uid: 36b266b1-2d70-497d-b88e-1d3089a26c84 [e2e-llm-inference-service] resourceVersion: '68428' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] openshift.io/internal-registry-pull-secret-ref: llmisvc-model-pvc-router-manage-2577e794-epp-sa-dockercfg-ktw7b [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: b4db031d-a9f8-4c49-a6d1-98105ac4d3e2 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: openshift.io/image-registry-pull-secrets_service-account-controller [e2e-llm-inference-service] operation: Apply [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:imagePullSecrets: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:openshift.io/internal-registry-pull-secret-ref: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] k:{"name":"llmisvc-model-pvc-router-manage-2577e794-epp-sa-dockercfg-ktw7b"}: {} [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b4db031d-a9f8-4c49-a6d1-98105ac4d3e2"}: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"default-dockercfg-lfhh2"}: {} [e2e-llm-inference-service] k:{"name":"seaweedfs-s3-creds"}: {} [e2e-llm-inference-service] secrets: [e2e-llm-inference-service] - name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: default-dockercfg-lfhh2 [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-2577e794-epp-sa-dockercfg-ktw7b [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-lfhh2 [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-2577e794-epp-sa-dockercfg-ktw7b [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: ServiceAccount [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] uid: 82571504-fb17-4801-89cd-a2d8aadc85fb [e2e-llm-inference-service] resourceVersion: '68406' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] openshift.io/internal-registry-pull-secret-ref: llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-zzdrr [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: b4db031d-a9f8-4c49-a6d1-98105ac4d3e2 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: openshift.io/image-registry-pull-secrets_service-account-controller [e2e-llm-inference-service] operation: Apply [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:imagePullSecrets: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:openshift.io/internal-registry-pull-secret-ref: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] k:{"name":"llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-zzdrr"}: {} [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b4db031d-a9f8-4c49-a6d1-98105ac4d3e2"}: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"default-dockercfg-lfhh2"}: {} [e2e-llm-inference-service] k:{"name":"seaweedfs-s3-creds"}: {} [e2e-llm-inference-service] secrets: [e2e-llm-inference-service] - name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: default-dockercfg-lfhh2 [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-zzdrr [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-lfhh2 [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-zzdrr [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: ServiceAccount [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-service [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] uid: f5708419-38d6-406f-a165-fa92049f709a [e2e-llm-inference-service] resourceVersion: '68472' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: b4db031d-a9f8-4c49-a6d1-98105ac4d3e2 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b4db031d-a9f8-4c49-a6d1-98105ac4d3e2"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] targetPort: grpc [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] targetPort: grpc-health [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] targetPort: metrics [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] targetPort: zmq [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] clusterIP: 172.31.242.9 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.242.9 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] uid: 3005486e-ef81-468e-9d71-99e9a5899563 [e2e-llm-inference-service] resourceVersion: '68417' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: b4db031d-a9f8-4c49-a6d1-98105ac4d3e2 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b4db031d-a9f8-4c49-a6d1-98105ac4d3e2"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] targetPort: 8000 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] clusterIP: 172.31.108.196 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.108.196 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-74696b7c6f [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] uid: 8886160c-3aa2-40d1-b66a-86d0f0b432ae [e2e-llm-inference-service] resourceVersion: '68444' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-worker [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] controller.kubernetes.io/hash: 74696b7c6f [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/group-index: '0' [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/group-key: ae449cef3b581b6dbab6ee0440d253929da5cb58 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/template-revision-hash: 7bc46f896d [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: StatefulSet [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 [e2e-llm-inference-service] uid: 7fec6a81-d121-42f6-93fe-7a509a63285d [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:data: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:controller.kubernetes.io/hash: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/group-index: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/group-key: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/name: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/template-revision-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7fec6a81-d121-42f6-93fe-7a509a63285d"}: {} [e2e-llm-inference-service] f:revision: {} [e2e-llm-inference-service] data: [e2e-llm-inference-service] spec: [e2e-llm-inference-service] template: [e2e-llm-inference-service] $patch: replace [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/leader-name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/size: '2' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-worker [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/group-index: '0' [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/group-key: ae449cef3b581b6dbab6ee0440d253929da5cb58 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/template-revision-hash: 7bc46f896d [e2e-llm-inference-service] spec: [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - args: [e2e-llm-inference-service] - --served-model-name [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --enable-ssl-refresh [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] command: [e2e-llm-inference-service] - vllm [e2e-llm-inference-service] - serve [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] name: main [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] add: [e2e-llm-inference-service] - IPC_LOCK [e2e-llm-inference-service] - SYS_RAWIO [e2e-llm-inference-service] - NET_RAW [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - mountPath: /home [e2e-llm-inference-service] name: home [e2e-llm-inference-service] - mountPath: /tmp [e2e-llm-inference-service] name: tmp-dir [e2e-llm-inference-service] - mountPath: /dev/shm [e2e-llm-inference-service] name: dshm [e2e-llm-inference-service] - mountPath: /models [e2e-llm-inference-service] name: model-cache [e2e-llm-inference-service] - mountPath: /var/run/kserve/tls [e2e-llm-inference-service] name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] - mountPath: /mnt/models [e2e-llm-inference-service] name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: home [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: tmp-dir [e2e-llm-inference-service] - emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 8Gi [e2e-llm-inference-service] name: dshm [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: model-cache [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] secretName: llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] revision: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ControllerRevision [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-5cf8fb667d [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] uid: 4f6b2168-a034-4cf8-a091-158ab3b49c12 [e2e-llm-inference-service] resourceVersion: '68420' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-leader [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] controller.kubernetes.io/hash: 5cf8fb667d [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/template-revision-hash: 7bc46f896d [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/worker-index: '0' [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/replicas: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: StatefulSet [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] uid: b2882710-1544-4efd-82d3-b54864075665 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:data: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/replicas: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:controller.kubernetes.io/hash: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/name: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/template-revision-hash: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/worker-index: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b2882710-1544-4efd-82d3-b54864075665"}: {} [e2e-llm-inference-service] f:revision: {} [e2e-llm-inference-service] data: [e2e-llm-inference-service] spec: [e2e-llm-inference-service] template: [e2e-llm-inference-service] $patch: replace [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/size: '2' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-leader [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/template-revision-hash: 7bc46f896d [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/worker-index: '0' [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] spec: [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - args: [e2e-llm-inference-service] - --served-model-name [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --enable-ssl-refresh [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] command: [e2e-llm-inference-service] - vllm [e2e-llm-inference-service] - serve [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] name: main [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] add: [e2e-llm-inference-service] - IPC_LOCK [e2e-llm-inference-service] - SYS_RAWIO [e2e-llm-inference-service] - NET_RAW [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - mountPath: /home [e2e-llm-inference-service] name: home [e2e-llm-inference-service] - mountPath: /tmp [e2e-llm-inference-service] name: tmp-dir [e2e-llm-inference-service] - mountPath: /dev/shm [e2e-llm-inference-service] name: dshm [e2e-llm-inference-service] - mountPath: /models [e2e-llm-inference-service] name: model-cache [e2e-llm-inference-service] - mountPath: /var/run/kserve/tls [e2e-llm-inference-service] name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] - mountPath: /mnt/models [e2e-llm-inference-service] name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: home [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: tmp-dir [e2e-llm-inference-service] - emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 8Gi [e2e-llm-inference-service] name: dshm [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: model-cache [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] secretName: llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] revision: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ControllerRevision [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheduler [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] uid: a6ee08f1-d7b4-47d6-99c9-cc930aa6b2e4 [e2e-llm-inference-service] resourceVersion: '69044' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: b4db031d-a9f8-4c49-a6d1-98105ac4d3e2 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b4db031d-a9f8-4c49-a6d1-98105ac4d3e2"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T11:11:20Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.10.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: llm-d.ai/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n-\ [e2e-llm-inference-service] \ type: single-profile-handler\n- type: queue-scorer\n- type: kv-cache-utilization-scorer\n\ [e2e-llm-inference-service] - type: prefix-cache-scorer\n- type: no-hit-lru-scorer\n- type: max-score-picker\n\ [e2e-llm-inference-service] - parameters:\n scheme: https\n type: metrics-data-source\nschedulingProfiles:\n\ [e2e-llm-inference-service] - name: default\n plugins:\n - pluginRef: queue-scorer\n weight: 2\n\ [e2e-llm-inference-service] \ - pluginRef: kv-cache-utilization-scorer\n weight: 2\n - pluginRef:\ [e2e-llm-inference-service] \ prefix-cache-scorer\n weight: 3\n - pluginRef: no-hit-lru-scorer\n\ [e2e-llm-inference-service] \ weight: 2\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: Recreate [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-28T11:11:20Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T11:11:20Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-28T11:11:20Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheduler-66dcb4d65d" [e2e-llm-inference-service] has successfully progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheduler-66dcb4d65d [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] uid: 940e498a-aa18-4c19-b0da-15e8cca91d03 [e2e-llm-inference-service] resourceVersion: '69042' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 66dcb4d65d [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheduler [e2e-llm-inference-service] uid: a6ee08f1-d7b4-47d6-99c9-cc930aa6b2e4 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"a6ee08f1-d7b4-47d6-99c9-cc930aa6b2e4"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T11:11:20Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 66dcb4d65d [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 66dcb4d65d [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.10.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: llm-d.ai/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n-\ [e2e-llm-inference-service] \ type: single-profile-handler\n- type: queue-scorer\n- type: kv-cache-utilization-scorer\n\ [e2e-llm-inference-service] - type: prefix-cache-scorer\n- type: no-hit-lru-scorer\n- type: max-score-picker\n\ [e2e-llm-inference-service] - parameters:\n scheme: https\n type: metrics-data-source\nschedulingProfiles:\n\ [e2e-llm-inference-service] - name: default\n plugins:\n - pluginRef: queue-scorer\n weight: 2\n\ [e2e-llm-inference-service] \ - pluginRef: kv-cache-utilization-scorer\n weight: 2\n - pluginRef:\ [e2e-llm-inference-service] \ prefix-cache-scorer\n weight: 3\n - pluginRef: no-hit-lru-scorer\n\ [e2e-llm-inference-service] \ weight: 2\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-rb [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] uid: d4139224-96ab-4bfc-98b5-0a9a7b5d7a6c [e2e-llm-inference-service] resourceVersion: '68466' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: b4db031d-a9f8-4c49-a6d1-98105ac4d3e2 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b4db031d-a9f8-4c49-a6d1-98105ac4d3e2"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] apiGroup: rbac.authorization.k8s.io [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-role [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-scc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] uid: 26b05267-148f-4127-837d-3bc965c43b67 [e2e-llm-inference-service] resourceVersion: '68403' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: b4db031d-a9f8-4c49-a6d1-98105ac4d3e2 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b4db031d-a9f8-4c49-a6d1-98105ac4d3e2"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-prefill [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] apiGroup: rbac.authorization.k8s.io [e2e-llm-inference-service] kind: ClusterRole [e2e-llm-inference-service] name: openshift-ai-llminferenceservice-scc [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-role [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] uid: 7cc572cb-fe19-4d3e-b8b3-6138f9053acb [e2e-llm-inference-service] resourceVersion: '68458' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: b4db031d-a9f8-4c49-a6d1-98105ac4d3e2 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b4db031d-a9f8-4c49-a6d1-98105ac4d3e2"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - llm-d.ai [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] - create [e2e-llm-inference-service] - update [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - delete [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-service-vw8q7 [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-2577e794-epp-service- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] uid: 8ec8b288-2b6f-452f-8a22-5513074e88aa [e2e-llm-inference-service] resourceVersion: '69041' [e2e-llm-inference-service] generation: 3 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: llmisvc-model-pvc-router-manage-2577e794-epp-service [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-28T11:11:20Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-service [e2e-llm-inference-service] uid: f5708419-38d6-406f-a165-fa92049f709a [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T11:11:20Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"f5708419-38d6-406f-a165-fa92049f709a"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.134.0.67 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] name: llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schekxjwb [e2e-llm-inference-service] uid: be54b512-094b-4f12-94ae-858699835a9e [e2e-llm-inference-service] nodeName: ip-10-0-141-50.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-s89rjl [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] uid: 72558f9e-002c-4d61-b797-c3853711221e [e2e-llm-inference-service] resourceVersion: '68422' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] uid: 3005486e-ef81-468e-9d71-99e9a5899563 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"3005486e-ef81-468e-9d71-99e9a5899563"}: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: null [e2e-llm-inference-service] ports: null [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-rb [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] uid: d4139224-96ab-4bfc-98b5-0a9a7b5d7a6c [e2e-llm-inference-service] resourceVersion: '68466' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: b4db031d-a9f8-4c49-a6d1-98105ac4d3e2 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b4db031d-a9f8-4c49-a6d1-98105ac4d3e2"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] userNames: [e2e-llm-inference-service] - system:serviceaccount:e2e-test-llm-inference-service-7c3633d1:llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] groupNames: null [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-role [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-scc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] uid: 26b05267-148f-4127-837d-3bc965c43b67 [e2e-llm-inference-service] resourceVersion: '68403' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: b4db031d-a9f8-4c49-a6d1-98105ac4d3e2 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b4db031d-a9f8-4c49-a6d1-98105ac4d3e2"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] userNames: [e2e-llm-inference-service] - system:serviceaccount:e2e-test-llm-inference-service-7c3633d1:llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] - system:serviceaccount:e2e-test-llm-inference-service-7c3633d1:llmisvc-model-pvc-router-manage-2577e794-kserve-mn-prefill [e2e-llm-inference-service] groupNames: null [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-prefill [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] name: openshift-ai-llminferenceservice-scc [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-role [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] uid: 7cc572cb-fe19-4d3e-b8b3-6138f9053acb [e2e-llm-inference-service] resourceVersion: '68458' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: b4db031d-a9f8-4c49-a6d1-98105ac4d3e2 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b4db031d-a9f8-4c49-a6d1-98105ac4d3e2"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - llm-d.ai [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - create [e2e-llm-inference-service] - delete [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - update [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b4db031d-a9f8-4c49-a6d1-98105ac4d3e2"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T11:11:00Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-28T11:11:00Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-28T11:11:01Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-route [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: b4db031d-a9f8-4c49-a6d1-98105ac4d3e2 [e2e-llm-inference-service] resourceVersion: '68829' [e2e-llm-inference-service] uid: 2e11e0fa-d990-4283-b18e-5868eebc9480 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T11:11:00Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T11:10:49Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b4db031d-a9f8-4c49-a6d1-98105ac4d3e2"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T11:11:00Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-28T11:11:00Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-28T11:11:01Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-route [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: b4db031d-a9f8-4c49-a6d1-98105ac4d3e2 [e2e-llm-inference-service] resourceVersion: '68829' [e2e-llm-inference-service] uid: 2e11e0fa-d990-4283-b18e-5868eebc9480 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T11:11:00Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T11:10:49Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b4db031d-a9f8-4c49-a6d1-98105ac4d3e2"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:endpointPickerRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:number: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:matchLabels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPorts: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-28T11:11:00Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: b4db031d-a9f8-4c49-a6d1-98105ac4d3e2 [e2e-llm-inference-service] resourceVersion: '68806' [e2e-llm-inference-service] uid: 386cb4d9-3286-4e95-ade2-99effecfb8e1 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] appProtocol: http [e2e-llm-inference-service] endpointPickerRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-service [e2e-llm-inference-service] port: [e2e-llm-inference-service] number: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPorts: [e2e-llm-inference-service] - number: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T11:11:00Z' [e2e-llm-inference-service] message: Referenced by an HTTPRoute accepted by the parentRef Gateway [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T11:11:00Z' [e2e-llm-inference-service] message: Referenced ExtensionRef resolved successfully [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: networking.istio.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] kind: AuthPolicy [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T11:10:49Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-policies [e2e-llm-inference-service] app.kubernetes.io/managed-by: odh-model-controller [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b4db031d-a9f8-4c49-a6d1-98105ac4d3e2"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:rules: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:authentication: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:public: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:anonymous: {} [e2e-llm-inference-service] f:credentials: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:overrides: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fairness: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:response: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:success: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:headers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:x-gateway-inference-fairness-id: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:x-gateway-inference-objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:targetRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T11:10:49Z' [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Accepted"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Enforced"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-28T11:10:51Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-route-authn [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: b4db031d-a9f8-4c49-a6d1-98105ac4d3e2 [e2e-llm-inference-service] resourceVersion: '68662' [e2e-llm-inference-service] uid: 21dae929-5285-4746-b626-1dcf9e0cf6d0 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] rules: [e2e-llm-inference-service] authentication: [e2e-llm-inference-service] public: [e2e-llm-inference-service] anonymous: {} [e2e-llm-inference-service] credentials: {} [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] overrides: [e2e-llm-inference-service] fairness: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] objective: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] response: [e2e-llm-inference-service] success: [e2e-llm-inference-service] headers: [e2e-llm-inference-service] x-gateway-inference-fairness-id: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.fairness [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] x-gateway-inference-objective: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.objective [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-route [e2e-llm-inference-service] status: [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T11:10:50Z' [e2e-llm-inference-service] message: AuthPolicy has been accepted [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T11:10:51Z' [e2e-llm-inference-service] message: AuthPolicy has been successfully enforced [e2e-llm-inference-service] reason: Enforced [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Enforced [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: leaderworkerset.x-k8s.io/v1 [e2e-llm-inference-service] kind: LeaderWorkerSet [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-worker [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: leaderworkerset.x-k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:conditions: {} [e2e-llm-inference-service] f:hpaPodSelector: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] manager: lws [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] - apiVersion: leaderworkerset.x-k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b4db031d-a9f8-4c49-a6d1-98105ac4d3e2"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:leaderWorkerTemplate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:leaderTemplate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:add: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:size: {} [e2e-llm-inference-service] f:workerTemplate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:add: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:rolloutStrategy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupPolicy: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: b4db031d-a9f8-4c49-a6d1-98105ac4d3e2 [e2e-llm-inference-service] resourceVersion: '68450' [e2e-llm-inference-service] uid: 903ac0ff-5fe4-45a0-8deb-e4177a2dfde1 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] leaderWorkerTemplate: [e2e-llm-inference-service] leaderTemplate: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-leader [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] spec: [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - args: [e2e-llm-inference-service] - --served-model-name [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --enable-ssl-refresh [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] command: [e2e-llm-inference-service] - vllm [e2e-llm-inference-service] - serve [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] name: main [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] add: [e2e-llm-inference-service] - IPC_LOCK [e2e-llm-inference-service] - SYS_RAWIO [e2e-llm-inference-service] - NET_RAW [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - mountPath: /home [e2e-llm-inference-service] name: home [e2e-llm-inference-service] - mountPath: /tmp [e2e-llm-inference-service] name: tmp-dir [e2e-llm-inference-service] - mountPath: /dev/shm [e2e-llm-inference-service] name: dshm [e2e-llm-inference-service] - mountPath: /models [e2e-llm-inference-service] name: model-cache [e2e-llm-inference-service] - mountPath: /var/run/kserve/tls [e2e-llm-inference-service] name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] - mountPath: /mnt/models [e2e-llm-inference-service] name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: home [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: tmp-dir [e2e-llm-inference-service] - emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 8Gi [e2e-llm-inference-service] name: dshm [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: model-cache [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] restartPolicy: RecreateGroupOnPodRestart [e2e-llm-inference-service] size: 2 [e2e-llm-inference-service] workerTemplate: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-worker [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] spec: [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - args: [e2e-llm-inference-service] - --served-model-name [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --enable-ssl-refresh [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] command: [e2e-llm-inference-service] - vllm [e2e-llm-inference-service] - serve [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] name: main [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] add: [e2e-llm-inference-service] - IPC_LOCK [e2e-llm-inference-service] - SYS_RAWIO [e2e-llm-inference-service] - NET_RAW [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - mountPath: /home [e2e-llm-inference-service] name: home [e2e-llm-inference-service] - mountPath: /tmp [e2e-llm-inference-service] name: tmp-dir [e2e-llm-inference-service] - mountPath: /dev/shm [e2e-llm-inference-service] name: dshm [e2e-llm-inference-service] - mountPath: /models [e2e-llm-inference-service] name: model-cache [e2e-llm-inference-service] - mountPath: /var/run/kserve/tls [e2e-llm-inference-service] name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] - mountPath: /mnt/models [e2e-llm-inference-service] name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: home [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: tmp-dir [e2e-llm-inference-service] - emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 8Gi [e2e-llm-inference-service] name: dshm [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: model-cache [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] networkConfig: [e2e-llm-inference-service] subdomainPolicy: Shared [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] rolloutStrategy: [e2e-llm-inference-service] rollingUpdateConfiguration: [e2e-llm-inference-service] maxSurge: 0 [e2e-llm-inference-service] maxUnavailable: 1 [e2e-llm-inference-service] partition: 0 [e2e-llm-inference-service] type: RollingUpdate [e2e-llm-inference-service] startupPolicy: LeaderCreated [e2e-llm-inference-service] status: [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] message: Replicas are progressing [e2e-llm-inference-service] reason: GroupsProgressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Progressing [e2e-llm-inference-service] hpaPodSelector: leaderworkerset.sigs.k8s.io/name=llmisvc-model-pvc-router-manage-2577e794-kserve-mn,leaderworkerset.sigs.k8s.io/worker-index=0 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b4db031d-a9f8-4c49-a6d1-98105ac4d3e2"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-scheduler [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: b4db031d-a9f8-4c49-a6d1-98105ac4d3e2 [e2e-llm-inference-service] resourceVersion: '68500' [e2e-llm-inference-service] uid: 7c29280b-7165-4c58-9653-92213f4f4c30 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-2577e794-epp-service.e2e-test-llm-inference-service-7c3633d1.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-2577e794-epp-service.e2e-test-llm-inference-service-7c3633d1.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T11:11:16Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b4db031d-a9f8-4c49-a6d1-98105ac4d3e2"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T11:11:16Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-shadow-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: b4db031d-a9f8-4c49-a6d1-98105ac4d3e2 [e2e-llm-inference-service] resourceVersion: '68988' [e2e-llm-inference-service] uid: ed6e6aed-f39b-4598-bdb9-cb026f11c49c [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-2577e794-inference--ip-950f33a3.e2e-test-llm-inference-service-7c3633d1.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc.e2e-test-llm-inference-service-7c3633d1.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T11:10:49Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b4db031d-a9f8-4c49-a6d1-98105ac4d3e2"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T11:10:49Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: b4db031d-a9f8-4c49-a6d1-98105ac4d3e2 [e2e-llm-inference-service] resourceVersion: '68517' [e2e-llm-inference-service] uid: 6d07e521-ad6c-4fc4-bc72-587add56df00 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc.e2e-test-llm-inference-service-7c3633d1.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc.e2e-test-llm-inference-service-7c3633d1.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b4db031d-a9f8-4c49-a6d1-98105ac4d3e2"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-scheduler [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: b4db031d-a9f8-4c49-a6d1-98105ac4d3e2 [e2e-llm-inference-service] resourceVersion: '68500' [e2e-llm-inference-service] uid: 7c29280b-7165-4c58-9653-92213f4f4c30 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-2577e794-epp-service.e2e-test-llm-inference-service-7c3633d1.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-2577e794-epp-service.e2e-test-llm-inference-service-7c3633d1.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T11:11:16Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b4db031d-a9f8-4c49-a6d1-98105ac4d3e2"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T11:11:16Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-shadow-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: b4db031d-a9f8-4c49-a6d1-98105ac4d3e2 [e2e-llm-inference-service] resourceVersion: '68988' [e2e-llm-inference-service] uid: ed6e6aed-f39b-4598-bdb9-cb026f11c49c [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-2577e794-inference--ip-950f33a3.e2e-test-llm-inference-service-7c3633d1.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc.e2e-test-llm-inference-service-7c3633d1.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T11:10:49Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b4db031d-a9f8-4c49-a6d1-98105ac4d3e2"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T11:10:49Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: b4db031d-a9f8-4c49-a6d1-98105ac4d3e2 [e2e-llm-inference-service] resourceVersion: '68517' [e2e-llm-inference-service] uid: 6d07e521-ad6c-4fc4-bc72-587add56df00 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc.e2e-test-llm-inference-service-7c3633d1.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc.e2e-test-llm-inference-service-7c3633d1.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b4db031d-a9f8-4c49-a6d1-98105ac4d3e2"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-scheduler [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: b4db031d-a9f8-4c49-a6d1-98105ac4d3e2 [e2e-llm-inference-service] resourceVersion: '68500' [e2e-llm-inference-service] uid: 7c29280b-7165-4c58-9653-92213f4f4c30 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-2577e794-epp-service.e2e-test-llm-inference-service-7c3633d1.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-2577e794-epp-service.e2e-test-llm-inference-service-7c3633d1.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T11:11:16Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b4db031d-a9f8-4c49-a6d1-98105ac4d3e2"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T11:11:16Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-shadow-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: b4db031d-a9f8-4c49-a6d1-98105ac4d3e2 [e2e-llm-inference-service] resourceVersion: '68988' [e2e-llm-inference-service] uid: ed6e6aed-f39b-4598-bdb9-cb026f11c49c [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-2577e794-inference--ip-950f33a3.e2e-test-llm-inference-service-7c3633d1.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc.e2e-test-llm-inference-service-7c3633d1.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T11:10:49Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b4db031d-a9f8-4c49-a6d1-98105ac4d3e2"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T11:10:49Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: b4db031d-a9f8-4c49-a6d1-98105ac4d3e2 [e2e-llm-inference-service] resourceVersion: '68517' [e2e-llm-inference-service] uid: 6d07e521-ad6c-4fc4-bc72-587add56df00 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc.e2e-test-llm-inference-service-7c3633d1.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc.e2e-test-llm-inference-service-7c3633d1.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b4db031d-a9f8-4c49-a6d1-98105ac4d3e2"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:extensionRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:portNumber: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPortNumber: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T11:10:48Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: b4db031d-a9f8-4c49-a6d1-98105ac4d3e2 [e2e-llm-inference-service] resourceVersion: '68492' [e2e-llm-inference-service] uid: 3e93bcb7-e23a-4393-af0e-83d963a838f5 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] extensionRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-service [e2e-llm-inference-service] portNumber: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPortNumber: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parent: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '1970-01-01T00:00:00Z' [e2e-llm-inference-service] message: Waiting for controller [e2e-llm-inference-service] reason: Pending [e2e-llm-inference-service] status: Unknown [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Status [e2e-llm-inference-service] name: default [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1 [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T11:25:31Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-worker [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] apps.kubernetes.io/pod-index: '1' [e2e-llm-inference-service] controller-revision-hash: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-74696b7c6f [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/group-index: '0' [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/group-key: ae449cef3b581b6dbab6ee0440d253929da5cb58 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/template-revision-hash: 7bc46f896d [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/worker-index: '1' [e2e-llm-inference-service] statefulset.kubernetes.io/pod-name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1 [e2e-llm-inference-service] timestamp: '2026-07-28T11:25:15Z' [e2e-llm-inference-service] window: 18.691s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 4394360n [e2e-llm-inference-service] memory: 2418088Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schekxjwb [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T11:25:31Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 66dcb4d65d [e2e-llm-inference-service] timestamp: '2026-07-28T11:25:10Z' [e2e-llm-inference-service] window: 16.597s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 596794n [e2e-llm-inference-service] memory: 22280Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_inference_service] [2026-07-28T11:25:32.149485] end - ❌ 902.477s: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T11:11:16Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T11:11:01Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T11:10:49Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T11:11:34Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T11:11:34Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T11:11:34Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T11:10:49Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T11:10:49Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ---------------------------- Captured log teardown ----------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.conftest:conftest.py:168 Skipping deletion of namespace e2e-test-llm-inference-service-7c3633d1 (SKIP_DELETION_ON_FAILURE) [e2e-llm-inference-service] _ test_llm_autoscaling_keda_lws[router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-keda] _ [e2e-llm-inference-service] [gw0] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-lws', 'prometheus-scrape', 'scaling-keda'], prompt='KSe... {'name': 'scaling-keda-autoscale-keda-lws-1337f511'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.autoscaling_keda [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator-lws", [e2e-llm-inference-service] "prometheus-scrape", [e2e-llm-inference-service] "scaling-keda", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="autoscale-keda-lws", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_multi_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_autoscaling_keda_lws(test_case: TestCase): [e2e-llm-inference-service] """KEDA + LWS: ScaledObject exists with WVA annotations; pods scale under load.""" [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] kserve_client = _new_kserve_client() [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] ns = test_case.namespace [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > _create_and_wait(kserve_client, test_case) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:728: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-lws', 'prometheus-scrape', 'scaling-keda'], prompt='KSe... {'name': 'scaling-keda-autoscale-keda-lws-1337f511'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] def _create_and_wait(kserve_client, test_case): [e2e-llm-inference-service] """Create LLMISVC and wait for it to be ready.""" [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:482: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...e-ked-231d315d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-lws-1337f511'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-07-28T11:21:03.229256', start_time = 1785237663.2295237 [e2e-llm-inference-service] duration = 900.1246600151062, timestamp_end = '2026-07-28T11:36:03.354187' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....toscale-ked-231d315d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-lws-1337f511'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1376: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7ff792df4720> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1387: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T11:22:10Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T11:22:10Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T11:21:45Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T11:22:21Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T11:22:21Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T11:22:10Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T11:22:21Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T11:22:10Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T11:22:10Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1371: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:82 Created test namespace e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret seaweedfs-s3-creds from kserve-ci-e2e-test to e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret storage-config from kserve-ci-e2e-test to e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:178 Patched default SA in e2e-test-llm-autoscaling-keda-lws-e541a132 with secret seaweedfs-s3-creds [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:162 ConfigMap odh-kserve-custom-ca-bundle already exists in e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:159 Copied ConfigMap odh-trusted-ca-bundle from kserve-ci-e2e-test to e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig router-managed-autoscale-keda-l-78828c4a in namespace e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig router-managed-autoscale-keda-l-78828c4a [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig router-managed-autoscale-keda-l-78828c4a [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig workload-llmd-simulator-lws-aut-1696d0b7 in namespace e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig workload-llmd-simulator-lws-aut-1696d0b7 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig workload-llmd-simulator-lws-aut-1696d0b7 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig prometheus-scrape-autoscale-ked-231d315d in namespace e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig prometheus-scrape-autoscale-ked-231d315d [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig prometheus-scrape-autoscale-ked-231d315d [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig scaling-keda-autoscale-keda-lws-1337f511 in namespace e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig scaling-keda-autoscale-keda-lws-1337f511 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig scaling-keda-autoscale-keda-lws-1337f511 [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_autoscaling_keda_lws] [2026-07-28T11:21:02.859268] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-llmd-simulator-lws', 'prometheus-scrape', 'scaling-keda'], prompt='KServe is a', service_name='autoscale-keda-lws', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, namespace='e2e-test-llm-autoscaling-keda-lws-e541a132', before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-keda-lws', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-keda-lws-e541a132', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-keda-l-78828c4a'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-lws-aut-1696d0b7'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-ked-231d315d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-lws-1337f511'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-28T11:21:02.875043] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-keda-lws', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-keda-lws-e541a132', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-keda-l-78828c4a'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-lws-aut-1696d0b7'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-ked-231d315d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-lws-1337f511'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-28T11:21:03.229122] end - ✅ in 0.353s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-28T11:21:03.229256] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-keda-lws', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-keda-lws-e541a132', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-keda-l-78828c4a'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-lws-aut-1696d0b7'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-ked-231d315d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-lws-1337f511'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'RouterReady', 'WorkloadsReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T11:21:45Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T11:21:45Z', 'message': 'Inference Pool e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T11:21:45Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T11:21:45Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T11:21:45Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T11:21:45Z', 'message': 'ScaledObject not yet visible in cache', 'reason': 'ScaledObjectProgressing', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T11:21:45Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T11:21:45Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T11:21:45Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'RouterReady', 'WorkloadsReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T11:22:10Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T11:22:10Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T11:21:45Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T11:22:10Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T11:22:10Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T11:22:10Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T11:22:10Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T11:22:10Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T11:22:10Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T11:22:10Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T11:22:10Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T11:21:45Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T11:22:21Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T11:22:21Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T11:22:10Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T11:22:21Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T11:22:10Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T11:22:10Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 12: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-28T10:22:24Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 13: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-28T10:22:24Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 14: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-28T10:22:24Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1391 Timed out waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T11:22:10Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T11:22:10Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T11:21:45Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T11:22:21Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T11:22:21Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T11:22:10Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T11:22:21Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T11:22:10Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T11:22:10Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-07-28T11:36:03.354187] end - ❌ 900.125s: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T11:22:10Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T11:22:10Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T11:21:45Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T11:22:21Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T11:22:21Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T11:22:10Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T11:22:21Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T11:22:10Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T11:22:10Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [delete_llmisvc] [2026-07-28T11:36:03.354364] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-keda-lws', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-keda-lws-e541a132', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-keda-l-78828c4a'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-lws-aut-1696d0b7'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-ked-231d315d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-lws-1337f511'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 3 pod(s) for autoscale-keda-lws still terminating: ['autoscale-keda-lws-kserve-mn-0', 'autoscale-keda-lws-kserve-mn-0-1', 'autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-t5ptm'] [e2e-llm-inference-service] assert not ['autoscale-keda-lws-kserve-mn-0', 'autoscale-keda-lws-kserve-mn-0-1', 'autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-t5ptm'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 1 pod(s) for autoscale-keda-lws still terminating: ['autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-t5ptm'] [e2e-llm-inference-service] assert not ['autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-t5ptm'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [delete_llmisvc] [2026-07-28T11:36:53.801942] end - ✅ in 50.447s [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_autoscaling_keda_lws] [2026-07-28T11:36:53.802067] end - ❌ 950.942s: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T11:22:10Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T11:22:10Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T11:21:45Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T11:22:21Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T11:22:21Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T11:22:10Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T11:22:21Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T11:22:10Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T11:22:10Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ---------------------------- Captured log teardown ----------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.conftest:conftest.py:168 Skipping deletion of namespace e2e-test-llm-autoscaling-keda-lws-e541a132 (SKIP_DELETION_ON_FAILURE) [e2e-llm-inference-service] _ test_llm_autoscaling_cleanup_hpa[router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-hpa] _ [e2e-llm-inference-service] [gw0] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-hpa'], prom... {'name': 'scaling-hpa-autoscale-cleanup-h-aa1ae037'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.autoscaling_hpa [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator-no-replicas", [e2e-llm-inference-service] "prometheus-scrape", [e2e-llm-inference-service] "scaling-hpa", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="autoscale-cleanup-hpa", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_autoscaling_cleanup_hpa(test_case: TestCase): [e2e-llm-inference-service] """Removing scaling config should delete HPA.""" [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] kserve_client = _new_kserve_client() [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] ns = test_case.namespace [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > _create_and_wait(kserve_client, test_case) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:892: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-hpa'], prom... {'name': 'scaling-hpa-autoscale-cleanup-h-aa1ae037'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] def _create_and_wait(kserve_client, test_case): [e2e-llm-inference-service] """Create LLMISVC and wait for it to be ready.""" [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:482: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...e-cle-5a67f5d1'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-cleanup-h-aa1ae037'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-07-28T11:36:54.931014', start_time = 1785238614.9312704 [e2e-llm-inference-service] duration = 900.5819141864777, timestamp_end = '2026-07-28T11:51:55.513186' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....toscale-cle-5a67f5d1'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-cleanup-h-aa1ae037'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1376: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7ff792df4680> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1387: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T11:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T11:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T11:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T11:37:30Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T11:38:03Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T11:38:03Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T11:38:03Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T11:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T11:38:03Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1371: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:82 Created test namespace e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret seaweedfs-s3-creds from kserve-ci-e2e-test to e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret storage-config from kserve-ci-e2e-test to e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:178 Patched default SA in e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40 with secret seaweedfs-s3-creds [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:162 ConfigMap odh-kserve-custom-ca-bundle already exists in e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:159 Copied ConfigMap odh-trusted-ca-bundle from kserve-ci-e2e-test to e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig router-managed-autoscale-cleanu-e5a6b97f in namespace e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig router-managed-autoscale-cleanu-e5a6b97f [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig router-managed-autoscale-cleanu-e5a6b97f [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig workload-llmd-simulator-no-repl-5d16e76d in namespace e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig workload-llmd-simulator-no-repl-5d16e76d [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig workload-llmd-simulator-no-repl-5d16e76d [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig prometheus-scrape-autoscale-cle-5a67f5d1 in namespace e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig prometheus-scrape-autoscale-cle-5a67f5d1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig prometheus-scrape-autoscale-cle-5a67f5d1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig scaling-hpa-autoscale-cleanup-h-aa1ae037 in namespace e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig scaling-hpa-autoscale-cleanup-h-aa1ae037 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig scaling-hpa-autoscale-cleanup-h-aa1ae037 [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_autoscaling_cleanup_hpa] [2026-07-28T11:36:54.681437] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-hpa'], prompt='KServe is a', service_name='autoscale-cleanup-hpa', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, namespace='e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40', before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-cleanup-hpa', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-cleanu-e5a6b97f'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-5d16e76d'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-cle-5a67f5d1'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-cleanup-h-aa1ae037'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-28T11:36:54.693959] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-cleanup-hpa', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-cleanu-e5a6b97f'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-5d16e76d'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-cle-5a67f5d1'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-cleanup-h-aa1ae037'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-28T11:36:54.930892] end - ✅ in 0.237s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-28T11:36:54.931014] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-cleanup-hpa', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-cleanu-e5a6b97f'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-5d16e76d'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-cle-5a67f5d1'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-cleanup-h-aa1ae037'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'RouterReady', 'WorkloadsReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T11:37:30Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T11:37:30Z', 'message': 'Inference Pool e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T11:37:30Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T11:37:30Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T11:37:30Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T11:37:30Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T11:37:30Z', 'message': 'HPA not yet visible in cache', 'reason': 'HPAProgressing', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T11:37:30Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T11:37:30Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 15: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-28T10:22:24Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T11:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T11:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T11:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T11:37:30Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T11:38:03Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T11:38:03Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T11:38:03Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T11:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T11:38:03Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 16: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-28T10:22:24Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 17: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-28T10:22:24Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1391 Timed out waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T11:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T11:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T11:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T11:37:30Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T11:38:03Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T11:38:03Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T11:38:03Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T11:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T11:38:03Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-07-28T11:51:55.513186] end - ❌ 900.582s: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T11:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T11:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T11:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T11:37:30Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T11:38:03Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T11:38:03Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T11:38:03Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T11:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T11:38:03Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [delete_llmisvc] [2026-07-28T11:51:55.513263] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-cleanup-hpa', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-cleanu-e5a6b97f'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-5d16e76d'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-cle-5a67f5d1'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-cleanup-h-aa1ae037'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 2 pod(s) for autoscale-cleanup-hpa still terminating: ['autoscale-cleanup-hpa-kserve-6cfdf55fc9-24z8h', 'autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-vmx75'] [e2e-llm-inference-service] assert not ['autoscale-cleanup-hpa-kserve-6cfdf55fc9-24z8h', 'autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-vmx75'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 1 pod(s) for autoscale-cleanup-hpa still terminating: ['autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-vmx75'] [e2e-llm-inference-service] assert not ['autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-vmx75'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [delete_llmisvc] [2026-07-28T11:52:20.634887] end - ✅ in 25.121s [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_autoscaling_cleanup_hpa] [2026-07-28T11:52:20.635012] end - ❌ 925.953s: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T11:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T11:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T11:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T11:37:30Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T11:38:03Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T11:38:03Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T11:38:03Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T11:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T11:38:03Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ---------------------------- Captured log teardown ----------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.conftest:conftest.py:168 Skipping deletion of namespace e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40 (SKIP_DELETION_ON_FAILURE) [e2e-llm-inference-service] =============================== warnings summary =============================== [e2e-llm-inference-service] llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-utilization-detector] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] /workspace/source/python/kserve/.venv/lib64/python3.11/site-packages/pytest_asyncio/plugin.py:761: DeprecationWarning: The event_loop fixture provided by pytest-asyncio has been redefined in [e2e-llm-inference-service] /workspace/source/test/e2e/conftest.py:43 [e2e-llm-inference-service] Replacing the event_loop fixture with a custom implementation is deprecated [e2e-llm-inference-service] and will lead to errors in the future. [e2e-llm-inference-service] If you want to request an asyncio event loop with a scope other than function [e2e-llm-inference-service] scope, use the "scope" argument to the asyncio mark when marking the tests. [e2e-llm-inference-service] If you want to return different types of event loops, use the event_loop_policy [e2e-llm-inference-service] fixture. [e2e-llm-inference-service] [e2e-llm-inference-service] warnings.warn( [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-utilization-detector] [e2e-llm-inference-service] llmisvc/test_flow_control.py:47: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-concurrency-detector] [e2e-llm-inference-service] llmisvc/test_flow_control.py:47: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-inline-config-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-qwen2.5-0.5b] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-configmap-ref-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-replicas-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-custom-template-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-nonzero-threshold-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-tokenizer-kvcache-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator2] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service_stop.py::test_llm_stop_feature[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service_stop.py:40: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_tls.py::test_llm_tls_resources[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_tls.py:92: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html [e2e-llm-inference-service] - generated xml file: /workspace/artifacts-dir/junit_e2e-llminferenceservice_and_cluster_cpu_and_not_autoscaling_and_not_tracing.xml - [e2e-llm-inference-service] --------------------------------- JSON report ---------------------------------- [e2e-llm-inference-service] report saved to: /workspace/artifacts-dir/e2e_results-llminferenceservice_and_cluster_cpu_and_not_autoscaling_and_not_tracing.json [e2e-llm-inference-service] =========================== short test summary info ============================ [e2e-llm-inference-service] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_hpa_deployment[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_keda_deployment[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-keda] [e2e-llm-inference-service] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-pvc] [e2e-llm-inference-service] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_hpa_lws[cluster_cpu-cluster_multi_node-router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-pvc] [e2e-llm-inference-service] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_keda_lws[cluster_cpu-cluster_multi_node-router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-keda] [e2e-llm-inference-service] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_cleanup_hpa[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 7 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! [e2e-llm-inference-service] !!!!!!!!!!!! xdist.dsession.Interrupted: stopping after 5 failures !!!!!!!!!!!!! [e2e-llm-inference-service] ====== 7 failed, 42 passed, 3 skipped, 25 warnings in 5420.56s (1:30:20) ======= [must-gather] [must-gather ] OUT 2026-07-28T11:52:24.716676547Z Using must-gather plug-in image: quay.io/modh/must-gather:rhoai-2.24 [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: 9c77e1ff-462e-41e6-a513-a8204698bb2b [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.26" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [must-gather] [must-gather ] OUT 2026-07-28T11:52:24.764095411Z namespace/openshift-must-gather-47lr5 created [must-gather] [must-gather ] OUT 2026-07-28T11:52:24.772234376Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-n8lqj created [must-gather] [must-gather ] OUT 2026-07-28T11:52:24.803768767Z pod for plug-in image quay.io/modh/must-gather:rhoai-2.24 created [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:31.884400227Z [disk usage checker] Started [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:31.888046196Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:32.185658205Z Error from server (NotFound): namespaces "redhat-ods-operator" not found [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:32.361513423Z Wrote inspect data to must-gather. [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:32.361548071Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:32.361548071Z namespaces "redhat-ods-operator" not found [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:32.364269951Z Error getting logs from redhat-ods-operator [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:32.545559815Z Wrote inspect data to must-gather. [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:32.545592585Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:32.545592585Z namespaces "redhat-ods-monitoring" not found [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:32.547931133Z Error getting logs from redhat-ods-monitoring [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:32.726033993Z Wrote inspect data to must-gather. [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:32.726077719Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:32.726077719Z namespaces "redhat-ods-applications" not found [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:32.728500300Z Error getting logs from redhat-ods-applications [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:32.917468422Z Wrote inspect data to must-gather. [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:32.917500150Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:32.917500150Z namespaces "rhods-notebooks" not found [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:32.920024043Z Error getting logs from rhods-notebooks [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:33.095578396Z Wrote inspect data to must-gather. [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:33.095612263Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:33.095612263Z namespaces "rhoai-model-registries" not found [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:33.098131691Z Error getting logs from rhoai-model-registries [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:33.282741827Z Wrote inspect data to must-gather. [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:33.282774493Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:33.282774493Z namespaces "istio-system" not found [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:33.285077316Z Error getting logs from istio-system [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:33.469119132Z Wrote inspect data to must-gather. [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:33.469151655Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:33.469151655Z namespaces "knative-serving" not found [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:33.471450249Z Error getting logs from knative-serving [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:33.665472494Z Wrote inspect data to must-gather. [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:33.665509554Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:33.665509554Z namespaces "redhat-ods-applications-auth-provider" not found [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:33.667896919Z Error getting logs from redhat-ods-applications-auth-provider [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:34.025187749Z Wrote inspect data to must-gather. [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:34.331810208Z Wrote inspect data to must-gather. [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:34.479528606Z error: the server doesn't have a resource type "auths" [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:34.564939153Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:34.566855597Z Error collecting info from [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:34.708312729Z error: the server doesn't have a resource type "monitorings" [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:34.805941852Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:34.808438970Z Error collecting info from [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:34.959072110Z error: the server doesn't have a resource type "featuretrackers" [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:35.055137079Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:35.057380531Z Error collecting info from [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:35.205129564Z error: the server doesn't have a resource type "codeflares" [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:35.294417513Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:35.296608451Z Error collecting info from [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:35.438385005Z error: the server doesn't have a resource type "dashboards" [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:35.533746406Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:35.535894826Z Error collecting info from [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:35.692735257Z error: the server doesn't have a resource type "datasciencepipelines" [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:35.781451399Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:35.783858679Z Error collecting info from [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:35.939917009Z error: the server doesn't have a resource type "feastoperators" [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:36.028486143Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:36.031237627Z Error collecting info from [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:36.182346464Z error: the server doesn't have a resource type "kserves" [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:36.272493830Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:36.275024452Z Error collecting info from [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:36.426338995Z error: the server doesn't have a resource type "kueues" [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:36.517226873Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:36.519701236Z Error collecting info from [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:36.662043817Z error: the server doesn't have a resource type "modelcontrollers" [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:36.762020412Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:36.764393328Z Error collecting info from [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:36.892886865Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:36.912739458Z error: the server doesn't have a resource type "modelmeshservings" [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:37.011095845Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:37.013385531Z Error collecting info from [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:37.166681568Z error: the server doesn't have a resource type "modelregistries" [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:37.262132100Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:37.264431065Z Error collecting info from [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:37.420952171Z error: the server doesn't have a resource type "rays" [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:37.513228002Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:37.515433164Z Error collecting info from [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:37.662650300Z error: the server doesn't have a resource type "trainingoperators" [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:37.761504546Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:37.763573044Z Error collecting info from [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:37.911892694Z error: the server doesn't have a resource type "trustyais" [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:38.011534770Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:38.013764710Z Error collecting info from [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:38.171037829Z error: the server doesn't have a resource type "workbenches" [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:38.268688227Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:38.270919861Z Error collecting info from [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:38.414929263Z error: the server doesn't have a resource type "hardwareprofiles" [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:38.513236628Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:38.515770501Z Error collecting info from [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:38.653798848Z error: the server doesn't have a resource type "llamastackoperators" [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:38.744883751Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:38.747470195Z Error collecting info from [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:39.545915976Z error: the server doesn't have a resource type "predictors" [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:39.685779827Z error: the server doesn't have a resource type "localmodelnodegroups" [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:39.831920572Z error: the server doesn't have a resource type "smcp" [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:39.975757460Z error: the server doesn't have a resource type "smm" [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:40.115580463Z error: the server doesn't have a resource type "smmr" [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:40.775972029Z error: the server doesn't have a resource type "knativeservings" [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:40.921457468Z error: the server doesn't have a resource type "configurations" [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:41.063849867Z error: the server doesn't have a resource type "routes" [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:41.221567880Z error: the server doesn't have a resource type "services" [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:41.361920024Z error: the server doesn't have a resource type "revisions" [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:41.897785440Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:41.949602801Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:42.094237858Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:42.187436048Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:42.263934101Z Wrote inspect data to must-gather. [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:42.446320039Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:42.599464292Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:42.704938048Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:42.792609990Z Wrote inspect data to must-gather. [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:42.973183147Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:43.121047315Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:43.224342320Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:43.302701939Z Wrote inspect data to must-gather. [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:43.477727607Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:43.620948985Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:43.706821525Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:43.784730092Z Wrote inspect data to must-gather. [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:43.954604121Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:44.097293284Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:44.181637859Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:44.260662929Z Wrote inspect data to must-gather. [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:44.434277775Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:44.581745443Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:44.668534035Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:44.746141573Z Wrote inspect data to must-gather. [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:44.915159564Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:45.056320131Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:45.143786854Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:45.223044864Z Wrote inspect data to must-gather. [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:45.393454893Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:45.536517594Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:45.617149765Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:45.695559695Z Wrote inspect data to must-gather. [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:45.867177353Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:46.010763971Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:46.096233818Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:46.174200282Z Wrote inspect data to must-gather. [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:46.351541338Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:46.499289905Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:46.584828147Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:46.663589736Z Wrote inspect data to must-gather. [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:46.837047002Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:46.903432896Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:46.982110284Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:47.075967239Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:47.159624547Z Wrote inspect data to must-gather. [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:47.328506224Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:47.477393454Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:47.566027692Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:47.643874524Z Wrote inspect data to must-gather. [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:47.814724016Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:47.959142597Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:48.047157852Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:48.124464097Z Wrote inspect data to must-gather. [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:48.292807552Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:48.438702218Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:48.522096942Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:48.602105989Z Wrote inspect data to must-gather. [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:48.779111858Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:49.284890434Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:49.452041034Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:49.564798579Z Wrote inspect data to must-gather. [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:49.735092143Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:50.230170961Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:50.410211454Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:50.530955944Z Wrote inspect data to must-gather. [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:50.708840306Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:51.211226504Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:51.376240706Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:51.508213193Z Wrote inspect data to must-gather. [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:51.687976003Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:51.909162728Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:52.243793485Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:52.432617712Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:52.552854556Z Wrote inspect data to must-gather. [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:52.726592431Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:53.502712272Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:53.708596658Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:53.833413439Z Wrote inspect data to must-gather. [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:54.020300851Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:54.940656101Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:55.140512153Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:55.297515338Z Wrote inspect data to must-gather. [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:55.472714967Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:56.271027527Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:56.463785798Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:56.592875805Z Wrote inspect data to must-gather. [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:56.775504078Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:56.914109979Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:57.543907615Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:57.736399903Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:57.870497480Z Wrote inspect data to must-gather. [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:58.043701982Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:58.813830452Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:59.006498626Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:59.133236610Z Wrote inspect data to must-gather. [must-gather] [must-gather-786cj] POD 2026-07-28T11:52:59.315553679Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:00.074191719Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:00.260665602Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:00.386495513Z Wrote inspect data to must-gather. [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:00.564738425Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:01.330141493Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:01.516297479Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:01.646581744Z Wrote inspect data to must-gather. [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:01.821975431Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:01.919142984Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:02.592911641Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:02.777112947Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:02.902885381Z Wrote inspect data to must-gather. [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:03.081187037Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:03.867320113Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:04.072021484Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:04.200352911Z Wrote inspect data to must-gather. [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:04.378317632Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:05.164325972Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:05.359907410Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:05.488701867Z Wrote inspect data to must-gather. [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:05.671542616Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:06.432152798Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:06.623530972Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:06.747702656Z Wrote inspect data to must-gather. [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:06.920431443Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:06.924070707Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:07.687393652Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:07.877736573Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:07.999388732Z Wrote inspect data to must-gather. [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:08.190433770Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:08.972859221Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:09.164744260Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:09.291339945Z Wrote inspect data to must-gather. [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:09.467328985Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:10.222903992Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:10.416535904Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:10.541272770Z Wrote inspect data to must-gather. [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:10.719741201Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:11.474916614Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:11.674646157Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:11.801550726Z Wrote inspect data to must-gather. [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:11.928895199Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:11.981524737Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:12.736252607Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:12.926276955Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:13.049019921Z Wrote inspect data to must-gather. [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:13.228748936Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:13.994540067Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:14.203009813Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:14.332896646Z Wrote inspect data to must-gather. [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:14.509751335Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:15.310336327Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:15.501271418Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:15.630045252Z Wrote inspect data to must-gather. [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:15.814627885Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:16.569938400Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:16.752939940Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:16.874127844Z Wrote inspect data to must-gather. [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:16.934445580Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:17.056891985Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:17.840834089Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:18.030775237Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:18.154395641Z Wrote inspect data to must-gather. [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:18.351379235Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:19.123384563Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:19.313455845Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:19.450232642Z Wrote inspect data to must-gather. [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:19.625025744Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:20.389888239Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:20.570489330Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:20.691844100Z Wrote inspect data to must-gather. [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:20.875081876Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:21.673955701Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:21.861736012Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:21.939463918Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:21.991506391Z Wrote inspect data to must-gather. [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:22.174885083Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:22.941662402Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:23.134318689Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:23.270109808Z Wrote inspect data to must-gather. [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:23.442403317Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:24.266284535Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:24.479346121Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:24.607261358Z Wrote inspect data to must-gather. [must-gather] [must-gather-786cj] POD 2026-07-28T11:53:24.630166453Z Caches written to disk [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.638924661Z waiting for gather to complete [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.64279311Z downloading gather output [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.936304272Z receiving incremental file list [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.948460919Z ./ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.948558831Z aggregated-discovery-api.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.948990099Z aggregated-discovery-apis.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.949748475Z event-filter.html [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.950137473Z timestamp [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.950289426Z version [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.955945611Z cluster-scoped-resources/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.955965781Z cluster-scoped-resources/datasciencecluster.opendatahub.io/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.955973481Z cluster-scoped-resources/datasciencecluster.opendatahub.io/datascienceclusters/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.956070803Z cluster-scoped-resources/datasciencecluster.opendatahub.io/datascienceclusters/test-dsc.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.956226017Z cluster-scoped-resources/dscinitialization.opendatahub.io/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.956252247Z cluster-scoped-resources/dscinitialization.opendatahub.io/dscinitializations/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.956267887Z cluster-scoped-resources/dscinitialization.opendatahub.io/dscinitializations/test-dsci.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.95639806Z namespaces/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.95641391Z namespaces/kserve-ci-e2e-test/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.956474302Z namespaces/kserve-ci-e2e-test/kserve-ci-e2e-test.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.956558323Z namespaces/kserve-ci-e2e-test/apps.openshift.io/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.956654275Z namespaces/kserve-ci-e2e-test/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.956738757Z namespaces/kserve-ci-e2e-test/apps/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.956801038Z namespaces/kserve-ci-e2e-test/apps/daemonsets.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.95688351Z namespaces/kserve-ci-e2e-test/apps/deployments.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.956995322Z namespaces/kserve-ci-e2e-test/apps/replicasets.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.957087574Z namespaces/kserve-ci-e2e-test/apps/statefulsets.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.957133515Z namespaces/kserve-ci-e2e-test/autoscaling/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.957207077Z namespaces/kserve-ci-e2e-test/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.957250177Z namespaces/kserve-ci-e2e-test/batch/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.957315949Z namespaces/kserve-ci-e2e-test/batch/cronjobs.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.957426121Z namespaces/kserve-ci-e2e-test/batch/jobs.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.957462251Z namespaces/kserve-ci-e2e-test/build.openshift.io/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.957499432Z namespaces/kserve-ci-e2e-test/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.957608354Z namespaces/kserve-ci-e2e-test/build.openshift.io/builds.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.957648965Z namespaces/kserve-ci-e2e-test/core/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.957712467Z namespaces/kserve-ci-e2e-test/core/configmaps.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.957841719Z namespaces/kserve-ci-e2e-test/core/endpoints.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.957935311Z namespaces/kserve-ci-e2e-test/core/events.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.958027403Z namespaces/kserve-ci-e2e-test/core/persistentvolumeclaims.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.958127855Z namespaces/kserve-ci-e2e-test/core/pods.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.958220067Z namespaces/kserve-ci-e2e-test/core/replicationcontrollers.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.95837753Z namespaces/kserve-ci-e2e-test/core/secrets.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.958590454Z namespaces/kserve-ci-e2e-test/core/services.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.958650506Z namespaces/kserve-ci-e2e-test/discovery.k8s.io/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.958698297Z namespaces/kserve-ci-e2e-test/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.958737448Z namespaces/kserve-ci-e2e-test/image.openshift.io/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.958790299Z namespaces/kserve-ci-e2e-test/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.958822949Z namespaces/kserve-ci-e2e-test/k8s.ovn.org/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.95887149Z namespaces/kserve-ci-e2e-test/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.958952212Z namespaces/kserve-ci-e2e-test/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.958988743Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.959023833Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.959077274Z namespaces/kserve-ci-e2e-test/networking.k8s.io/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.959113425Z namespaces/kserve-ci-e2e-test/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.959168146Z namespaces/kserve-ci-e2e-test/policy/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.959187947Z namespaces/kserve-ci-e2e-test/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.959260838Z namespaces/kserve-ci-e2e-test/route.openshift.io/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.959298909Z namespaces/kserve-ci-e2e-test/route.openshift.io/routes.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.95935042Z namespaces/kuadrant-system/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.959389671Z namespaces/kuadrant-system/kuadrant-system.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.959460602Z namespaces/kuadrant-system/apps.openshift.io/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.959497503Z namespaces/kuadrant-system/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.959533604Z namespaces/kuadrant-system/apps/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.959591435Z namespaces/kuadrant-system/apps/daemonsets.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.959707257Z namespaces/kuadrant-system/apps/deployments.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.960090255Z namespaces/kuadrant-system/apps/replicasets.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.960425432Z namespaces/kuadrant-system/apps/statefulsets.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.960483653Z namespaces/kuadrant-system/autoscaling/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.960531134Z namespaces/kuadrant-system/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.960598685Z namespaces/kuadrant-system/batch/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.960677777Z namespaces/kuadrant-system/batch/cronjobs.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.960786099Z namespaces/kuadrant-system/batch/jobs.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.96083683Z namespaces/kuadrant-system/build.openshift.io/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.960876991Z namespaces/kuadrant-system/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.960957803Z namespaces/kuadrant-system/build.openshift.io/builds.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.960999203Z namespaces/kuadrant-system/core/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.961029504Z namespaces/kuadrant-system/core/configmaps.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.961211578Z namespaces/kuadrant-system/core/endpoints.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.96134152Z namespaces/kuadrant-system/core/events.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.962078975Z namespaces/kuadrant-system/core/persistentvolumeclaims.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.962171867Z namespaces/kuadrant-system/core/pods.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.962588326Z namespaces/kuadrant-system/core/replicationcontrollers.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.962736029Z namespaces/kuadrant-system/core/secrets.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.962966013Z namespaces/kuadrant-system/core/services.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.963106716Z namespaces/kuadrant-system/discovery.k8s.io/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.963151797Z namespaces/kuadrant-system/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.96329645Z namespaces/kuadrant-system/image.openshift.io/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.963340641Z namespaces/kuadrant-system/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.963408582Z namespaces/kuadrant-system/k8s.ovn.org/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.963451263Z namespaces/kuadrant-system/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.963552505Z namespaces/kuadrant-system/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.963616696Z namespaces/kuadrant-system/monitoring.coreos.com/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.963683998Z namespaces/kuadrant-system/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.963726069Z namespaces/kuadrant-system/networking.k8s.io/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.96378823Z namespaces/kuadrant-system/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.963839231Z namespaces/kuadrant-system/pods/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.963846141Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-5xmkm/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.963900072Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-5xmkm/authorino-7cf4765b85-5xmkm.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.963990874Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-5xmkm/authorino/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.963998064Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-5xmkm/authorino/authorino/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.964002615Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-5xmkm/authorino/authorino/logs/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.964051376Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-5xmkm/authorino/authorino/logs/current.log [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.979074061Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-5xmkm/authorino/authorino/logs/previous.insecure.log [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.979159832Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-5xmkm/authorino/authorino/logs/previous.log [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.979184283Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-tlmfp/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.979265934Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-tlmfp/authorino-operator-56fb6f6856-tlmfp.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.979370127Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-tlmfp/manager/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.979383567Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-tlmfp/manager/manager/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.979388057Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-tlmfp/manager/manager/logs/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.979429358Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-tlmfp/manager/manager/logs/current.log [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.979653902Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-tlmfp/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.979734394Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-tlmfp/manager/manager/logs/previous.log [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.979783675Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-8bdq9/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.979857366Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-8bdq9/dns-operator-controller-manager-ccbdf686-8bdq9.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.979962909Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-8bdq9/manager/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.979975979Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-8bdq9/manager/manager/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.979982039Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-8bdq9/manager/manager/logs/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.98002781Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-8bdq9/manager/manager/logs/current.log [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.980138992Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-8bdq9/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.980213454Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-8bdq9/manager/manager/logs/previous.log [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.980256975Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-srd6m/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.980329866Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-srd6m/kuadrant-console-plugin-598f86468b-srd6m.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.980429838Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-srd6m/kuadrant-console-plugin/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.980442378Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-srd6m/kuadrant-console-plugin/kuadrant-console-plugin/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.980449249Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-srd6m/kuadrant-console-plugin/kuadrant-console-plugin/logs/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.98049503Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-srd6m/kuadrant-console-plugin/kuadrant-console-plugin/logs/current.log [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.980585991Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-srd6m/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.insecure.log [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.980696714Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-srd6m/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.log [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.980727164Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-2z85v/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.980765815Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-2z85v/kuadrant-operator-controller-manager-66866f5fcc-2z85v.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.980904508Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-2z85v/manager/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.980912558Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-2z85v/manager/manager/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.980916598Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-2z85v/manager/manager/logs/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.980946799Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-2z85v/manager/manager/logs/current.log [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.993774309Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-2z85v/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.993853761Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-2z85v/manager/manager/logs/previous.log [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.993901962Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-2zlsl/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.993950643Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-2zlsl/limitador-limitador-85d9db4d5d-2zlsl.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.994054775Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-2zlsl/limitador/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.994064195Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-2zlsl/limitador/limitador/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.994069245Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-2zlsl/limitador/limitador/logs/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.994096876Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-2zlsl/limitador/limitador/logs/current.log [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.994212568Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-2zlsl/limitador/limitador/logs/previous.insecure.log [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.99429551Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-2zlsl/limitador/limitador/logs/previous.log [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.994343971Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-5grlb/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.994389402Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-5grlb/limitador-operator-controller-manager-774d94476f-5grlb.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.994494794Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-5grlb/manager/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.994503954Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-5grlb/manager/manager/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.994508864Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-5grlb/manager/manager/logs/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.994555285Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-5grlb/manager/manager/logs/current.log [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.99479076Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-5grlb/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.994873691Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-5grlb/manager/manager/logs/previous.log [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.994897002Z namespaces/kuadrant-system/policy/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.994964463Z namespaces/kuadrant-system/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.995014184Z namespaces/kuadrant-system/route.openshift.io/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.995081056Z namespaces/kuadrant-system/route.openshift.io/routes.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.995120867Z namespaces/openshift-ingress/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.995202188Z namespaces/openshift-ingress/openshift-ingress.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.99528011Z namespaces/openshift-ingress/apps.openshift.io/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.995347051Z namespaces/openshift-ingress/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.995394112Z namespaces/openshift-ingress/apps/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.995445083Z namespaces/openshift-ingress/apps/daemonsets.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.995541575Z namespaces/openshift-ingress/apps/deployments.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.995885242Z namespaces/openshift-ingress/apps/replicasets.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.996105167Z namespaces/openshift-ingress/apps/statefulsets.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.996140997Z namespaces/openshift-ingress/autoscaling/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.996188898Z namespaces/openshift-ingress/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.996242779Z namespaces/openshift-ingress/batch/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.99629266Z namespaces/openshift-ingress/batch/cronjobs.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.996377282Z namespaces/openshift-ingress/batch/jobs.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.996412103Z namespaces/openshift-ingress/build.openshift.io/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.996464004Z namespaces/openshift-ingress/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.996534005Z namespaces/openshift-ingress/build.openshift.io/builds.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.996599866Z namespaces/openshift-ingress/core/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.996670458Z namespaces/openshift-ingress/core/configmaps.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.997089416Z namespaces/openshift-ingress/core/endpoints.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.997204729Z namespaces/openshift-ingress/core/events.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.997420243Z namespaces/openshift-ingress/core/persistentvolumeclaims.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.997493665Z namespaces/openshift-ingress/core/pods.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.99777659Z namespaces/openshift-ingress/core/replicationcontrollers.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.997922853Z namespaces/openshift-ingress/core/secrets.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.998119488Z namespaces/openshift-ingress/core/services.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.998211739Z namespaces/openshift-ingress/discovery.k8s.io/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.99826251Z namespaces/openshift-ingress/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.998348202Z namespaces/openshift-ingress/image.openshift.io/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.998387953Z namespaces/openshift-ingress/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.998457524Z namespaces/openshift-ingress/k8s.ovn.org/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.998498415Z namespaces/openshift-ingress/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.998640948Z namespaces/openshift-ingress/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.998699059Z namespaces/openshift-ingress/monitoring.coreos.com/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.99875381Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.998818192Z namespaces/openshift-ingress/networking.k8s.io/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.998867183Z namespaces/openshift-ingress/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.998931514Z namespaces/openshift-ingress/pods/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.998942804Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-w6tmc/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.998983355Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-w6tmc/istiod-openshift-gateway-5677ccb49b-w6tmc.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.999087487Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-w6tmc/discovery/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.999095277Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-w6tmc/discovery/discovery/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.999099437Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-w6tmc/discovery/discovery/logs/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:25.999154569Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-w6tmc/discovery/discovery/logs/current.log [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:26.019184005Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-w6tmc/discovery/discovery/logs/previous.insecure.log [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:26.019239757Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-w6tmc/discovery/discovery/logs/previous.log [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:26.019282187Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:26.019343679Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:26.019484022Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf/istio-proxy/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:26.019495062Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf/istio-proxy/istio-proxy/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:26.019501352Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:26.019512092Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:26.021201576Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:26.021336389Z namespaces/openshift-ingress/pods/router-default-65b54959b5-rhznz/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:26.02137762Z namespaces/openshift-ingress/pods/router-default-65b54959b5-rhznz/router-default-65b54959b5-rhznz.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:26.021491062Z namespaces/openshift-ingress/pods/router-default-65b54959b5-rhznz/router/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:26.021500452Z namespaces/openshift-ingress/pods/router-default-65b54959b5-rhznz/router/router/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:26.021505012Z namespaces/openshift-ingress/pods/router-default-65b54959b5-rhznz/router/router/logs/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:26.021515953Z namespaces/openshift-ingress/pods/router-default-65b54959b5-rhznz/router/router/logs/current.log [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:26.021658686Z namespaces/openshift-ingress/pods/router-default-65b54959b5-rhznz/router/router/logs/previous.insecure.log [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:26.021731837Z namespaces/openshift-ingress/pods/router-default-65b54959b5-rhznz/router/router/logs/previous.log [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:26.021750397Z namespaces/openshift-ingress/policy/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:26.021802529Z namespaces/openshift-ingress/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:26.02185145Z namespaces/openshift-ingress/route.openshift.io/ [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:26.02188938Z namespaces/openshift-ingress/route.openshift.io/routes.yaml [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:26.029867492Z [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:26.029885383Z sent 2,840 bytes received 1,114,943 bytes 745,188.67 bytes/sec [must-gather] [must-gather-786cj] OUT 2026-07-28T11:53:26.029898613Z total size is 16,216,548 speedup is 14.51 [must-gather] [must-gather ] OUT 2026-07-28T11:53:26.082092173Z namespace/openshift-must-gather-47lr5 deleted [must-gather] [must-gather] [must-gather] Reprinting Cluster State: [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: 9c77e1ff-462e-41e6-a513-a8204698bb2b [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.26" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather ] OUT 2026-07-28T11:53:26.20498871Z Using must-gather plug-in image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:35f66492587d98a3cca9bf7ba4541c867770dc76b78b4cf9352e6715dca04502 [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: 9c77e1ff-462e-41e6-a513-a8204698bb2b [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.26" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [must-gather] [must-gather ] OUT 2026-07-28T11:53:26.228103279Z namespace/openshift-must-gather-wxwvv created [must-gather] [must-gather ] OUT 2026-07-28T11:53:26.236738084Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-vpkjs created [must-gather] [must-gather ] OUT 2026-07-28T11:53:26.261672451Z pod for plug-in image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:35f66492587d98a3cca9bf7ba4541c867770dc76b78b4cf9352e6715dca04502 created [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:27.781329780Z [disk usage checker] Started [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:27.784905844Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:28.003950268Z Gathering data for ns/openshift-cluster-version... [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:28.165742970Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:28.301544147Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:28.409877875Z Gathering data for ns/default... [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:28.784748211Z Gathering data for ns/openshift... [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:29.275041657Z Gathering data for ns/kube-system... [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:30.147107784Z Gathering data for ns/openshift-etcd... [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:30.533536405Z Wrote inspect data to must-gather. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:30.533575453Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:30.533575453Z namespaces "assisted-installer" not found [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:30.910331228Z Waiting on subprocesses to finish execution. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:30.917803003Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:30.920869228Z INFO: Gathering machine config daemon's old logs from all nodes [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:30.928627223Z INFO: Gathering on-disk MachineConfig from degraded nodes [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:30.932090509Z INFO: Gathering HAProxy config files [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:30.935576828Z Executing Istio gather script [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:30.936107911Z INFO: Collecting host service logs for crio [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:30.936520770Z INFO: Collecting host service logs for kubelet [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:30.936814791Z INFO: Collecting host service logs for rpm-ostreed [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:30.937095038Z INFO: Collecting host service logs for ostree-finalize-staged [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:30.937427028Z INFO: Collecting host service logs for machine-config-daemon-firstboot [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:30.937710308Z INFO: Collecting host service logs for machine-config-daemon-host [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:30.938025882Z INFO: Collecting host service logs for NetworkManager [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:30.938338862Z INFO: Collecting host service logs for openvswitch [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:30.938685574Z INFO: Collecting host service logs for ovs-configuration [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:30.939024272Z INFO: Collecting host service logs for ovsdb-server [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:30.939406341Z INFO: Collecting host service logs for ovs-vswitchd [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:30.939788999Z INFO: Waiting for worker host service log collection to complete ... [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:31.011496697Z INFO: Waiting for node performance related collection to complete ... [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:31.648004751Z No resources found in openshift-etcd namespace. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:31.809041322Z error: the server doesn't have a resource type "performanceprofile" [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:31.890926942Z INFO: Found 1 replicas - prometheus-k8s-0 [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:32.231696612Z INFO: namespace openshift-frr-k8s not detected. Skipping. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:32.274207511Z No resources found [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:32.377058995Z INFO: "sriov-network-operator" not detected. Skipping. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:32.377543959Z Wrote inspect data to must-gather. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:32.459024881Z INFO: "metallb-operator" not detected. Skipping. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:32.475606460Z INFO: "kubernetes-nmstate-operator" not detected. Skipping. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:32.486065346Z INFO: Waiting for on-disk MachineConfig collection to complete ... [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:32.486226803Z INFO: on-disk MachineConfig config collection complete. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:32.617603131Z INFO: Collecting Insights Archives from insights-operator-5885fdb759-r6ptx [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:32.617649799Z insights-runtime-extractor-mpcmc [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:32.617669656Z insights-runtime-extractor-mtcd6 [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:32.617689413Z insights-runtime-extractor-vj25w [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:32.730498460Z INFO: Worker host service log collection to complete. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:32.818594435Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:32.999380273Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:32.999380273Z See 'oc rsync -h' for help and examples [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:33.139533543Z INFO: Getting alertmanagers from prometheus-k8s-0 [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:33.331161555Z ERROR: No running kube-apiserver pods found [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:33.452390537Z INFO: 'previous-logs' folder not found on ip-10-0-130-235.ec2.internal, skipping... [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:33.630958894Z No resources found [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:33.959147389Z INFO: Waiting for HAProxy config collection to complete ... [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:34.001753316Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:34.174245988Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:34.253879810Z error: the server doesn't have a resource type "multi-networkpolicy" [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:34.268615595Z Wrote inspect data to must-gather. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:34.338541395Z Wrote inspect data to must-gather. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:34.341544813Z Wrote inspect data to must-gather. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:34.479753662Z Wrote inspect data to must-gather. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:34.492556425Z INFO: 'previous-logs' folder not found on ip-10-0-131-88.ec2.internal, skipping... [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:34.608122942Z tar: Removing leading `/' from member names [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:34.617610664Z INFO: HAProxy config collection complete. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:34.646503408Z Inspecting resource clusterrole.rbac.authorization.k8s.io/istio-reader-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:34.722158701Z INFO: Getting rules from prometheus-k8s-0 [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:34.940149745Z INFO: 'previous-logs' folder not found on ip-10-0-141-50.ec2.internal, skipping... [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:34.940271948Z INFO: Waiting for Machine Config Daemon termination log collection to complete ... [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:34.940331349Z INFO: Machine Config Daemon termination log collection complete. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:35.030454345Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:35.034297381Z Inspecting resource clusterrole.rbac.authorization.k8s.io/istiod-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:35.091921700Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:35.358662713Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:35.361571907Z Inspecting resource clusterrole.rbac.authorization.k8s.io/istiod-gateway-controller-openshift-gateway-openshift-ingress [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:35.386892542Z INFO: Getting status/config from prometheus-k8s-0 [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:35.452585710Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:35.666342265Z Gathering data for ns/openshift-monitoring... [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:35.913973463Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:37.555091824Z Wrote inspect data to must-gather. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:37.844560294Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:37.971412055Z Gathering data for ns/openshift-network-console... [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:38.426687254Z Gathering data for ns/openshift-console-operator... [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:38.836562796Z Gathering data for ns/openshift-console... [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:39.412729404Z Gathering data for ns/openshift-cluster-storage-operator... [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:39.881624611Z Wrote inspect data to must-gather. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:40.028820566Z Gathering data for ns/openshift-dns-operator... [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:40.056605308Z error: the server doesn't have a resource type "machineconfigs" [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:40.227009361Z error: the server doesn't have a resource type "machineconfigpools" [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:40.395415456Z Gathering data for ns/openshift-dns... [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:40.588386745Z Wrote inspect data to must-gather. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:40.760284632Z error: the server doesn't have a resource type "kubeletconfigs" [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:41.111243796Z Gathering data for ns/openshift-image-registry... [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:41.156108747Z Wrote inspect data to must-gather. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:41.512626349Z Wrote inspect data to must-gather. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:41.721966831Z Gathering data for ns/openshift-ingress-operator... [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:41.947334181Z Wrote inspect data to must-gather. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:42.124050119Z Wrote inspect data to must-gather. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:42.177114865Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:42.383795634Z Wrote inspect data to must-gather. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:42.554154659Z Wrote inspect data to must-gather. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:42.803101130Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:42.805893700Z Inspecting resource clusterrolebinding.rbac.authorization.k8s.io/istio-reader-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:42.817244614Z Wrote inspect data to must-gather. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:42.854576409Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:43.066087605Z INFO: Getting status/flags from prometheus-k8s-0 [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:43.088219726Z Wrote inspect data to must-gather. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:43.112939706Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:43.117110305Z Inspecting resource clusterrolebinding.rbac.authorization.k8s.io/istiod-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:43.219742241Z Gathering data for ns/openshift-ingress-canary... [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:43.365876804Z INFO: Getting status/runtimeinfo from prometheus-k8s-0 [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:43.388126301Z Wrote inspect data to must-gather. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:43.439112224Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:43.446644322Z Inspecting resource clusterrolebinding.rbac.authorization.k8s.io/istiod-gateway-controller-openshift-gateway-openshift-ingress [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:43.493197475Z INFO: INTERCONNECT MODE [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:43.493253303Z INFO: Gathering ovn-kubernetes DBs [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:43.656918518Z INFO: Getting targets?state=active from prometheus-k8s-0 [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:43.716740070Z INFO: Gathering OVN_Northbound from ovnkube-node-4sg48... [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:43.717088729Z INFO: Gathering OVN_Northbound from ovnkube-node-cjlcq... [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:43.717426449Z INFO: Gathering OVN_Northbound from ovnkube-node-hdqls... [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:43.718021643Z INFO: Gathering OVN_Southbound from ovnkube-node-4sg48... [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:43.718316674Z INFO: Gathering OVN_Southbound from ovnkube-node-cjlcq... [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:43.718642792Z INFO: Gathering OVN_Southbound from ovnkube-node-hdqls... [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:43.770803645Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:44.061153356Z tar: Removing leading `/' from member names [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:44.076178243Z Wrote inspect data to must-gather. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:44.092872968Z Gathering data for ns/openshift-insights... [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:44.120737429Z INFO: Getting status/tsdb from prometheus-k8s-0 [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:44.135670397Z tar: Removing leading `/' from member names [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:44.164046157Z tar: Removing leading `/' from member names [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:44.173957154Z tar: Removing leading `/' from member names [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:44.190947602Z tar: Removing leading `/' from member names [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:44.203981757Z tar: Removing leading `/' from member names [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:44.413778501Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:44.418135195Z INFO: Fallback to identify the container image from release info [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:44.628706623Z INFO: Getting status from alertmanager-main-0 [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:44.734838532Z INFO: Waiting for network log collection to complete ... [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:44.734942591Z INFO: Waiting for ovnk database copies to complete ... [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:44.735411004Z INFO: Copying ovnk databases complete. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:44.737321439Z 6.9M must-gather/network_logs/ovnk_database_store [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:44.739736514Z ovnk_database_store/ [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:44.739736514Z ovnk_database_store/ovnkube-node-4sg48_nbdb [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:44.755561802Z ovnk_database_store/ovnkube-node-cjlcq_nbdb [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:44.773350133Z ovnk_database_store/ovnkube-node-hdqls_sbdb [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:44.818148199Z ovnk_database_store/ovnkube-node-4sg48_sbdb [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:44.855224723Z ovnk_database_store/ovnkube-node-hdqls_nbdb [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:44.870796443Z ovnk_database_store/ovnkube-node-cjlcq_sbdb [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:44.919658857Z INFO: Network log collection complete. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:45.118779005Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811 [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:45.269451558Z daemonset.apps/perf-node-gather-daemonset created [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:45.418680928Z Waiting for performance profile collector pods to become ready: 1 [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:46.212142870Z Inspecting resource crd/authorizationpolicies.security.istio.io [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:46.398235086Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:46.579341147Z Waiting for performance profile collector pods to become ready: 2 [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:46.636096828Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:46.638711379Z Inspecting resource crd/destinationrules.networking.istio.io [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:46.872483054Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:47.097610946Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:47.100031435Z Inspecting resource crd/envoyfilters.networking.istio.io [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:47.300907398Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:47.399038886Z Gathering data for ns/openshift-lws-operator... [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:47.535505909Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:47.540138696Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:47.738157625Z Waiting for performance profile collector pods to become ready: 3 [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:47.759095203Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:47.862137121Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:48.013928239Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:48.016197928Z Inspecting resource crd/gateways.gateway.networking.k8s.io [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:48.078272443Z Gathering data for ns/kserve... [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:48.209847626Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:48.428693735Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:48.431255738Z Inspecting resource crd/gateways.networking.istio.io [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:48.632341044Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:48.855982380Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:48.858405473Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:48.887089164Z Waiting for performance profile collector pods to become ready: 4 [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:49.087814998Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:49.327730493Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:49.330261453Z Inspecting resource crd/httproutes.gateway.networking.k8s.io [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:49.571864864Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:49.796046726Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:49.798777393Z Inspecting resource crd/inferencepools.inference.networking.k8s.io [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:50.012902402Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:50.064875680Z Waiting for performance profile collector pods to become ready: 5 [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:50.251706584Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:50.254280642Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:50.456943412Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:50.508750142Z Gathering data for ns/openshift-config... [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:50.693159929Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:50.695868077Z Inspecting resource crd/peerauthentications.security.istio.io [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:50.880763243Z Gathering data for ns/openshift-config-managed... [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:50.921926334Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:51.155714766Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:51.158335241Z Inspecting resource crd/proxyconfigs.networking.istio.io [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:51.223503641Z Waiting for performance profile collector pods to become ready: 6 [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:51.365140438Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:51.431013847Z Gathering data for ns/openshift-kube-apiserver-operator... [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:51.600461568Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:51.602995123Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:51.782743100Z Gathering data for ns/openshift-kube-apiserver... [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:51.822955159Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:52.075615041Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:52.078699142Z Inspecting resource crd/requestauthentications.security.istio.io [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:52.161914767Z Gathering data for ns/openshift-kube-controller-manager... [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:52.304898888Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:52.388607441Z Waiting for performance profile collector pods to become ready: 7 [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:52.553307551Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:52.556026394Z Inspecting resource crd/serviceentries.networking.istio.io [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:52.575110624Z Gathering data for ns/openshift-kube-controller-manager-operator... [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:52.774027705Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:52.869792931Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:52.969098008Z Gathering data for ns/openshift-kube-scheduler... [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:53.030398789Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:53.034565477Z Inspecting resource crd/sidecars.networking.istio.io [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:53.256214836Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:53.325965247Z Gathering data for ns/openshift-kube-scheduler-operator... [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:53.509128263Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:53.512291050Z Inspecting resource crd/telemetries.telemetry.istio.io [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:53.549762493Z Waiting for performance profile collector pods to become ready: 8 [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:53.747708700Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:53.756017184Z Gathering data for ns/openshift-kube-storage-version-migrator... [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:53.979582483Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:53.982616425Z Inspecting resource crd/virtualservices.networking.istio.io [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:54.181221718Z Gathering data for ns/openshift-kube-storage-version-migrator-operator... [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:54.227700143Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:54.470876915Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:54.473934104Z Inspecting resource crd/wasmplugins.extensions.istio.io [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:54.581541242Z Gathering data for ns/openshift-user-workload-monitoring... [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:54.710744733Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:54.714841586Z Daemonset perf-node-gather-daemonset ready 3 out of 3 [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:54.987077335Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:54.990609674Z Inspecting resource crd/workloadentries.networking.istio.io [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:55.063203644Z Collecting performance related data for node ip-10-0-141-50.ec2.internal [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:55.070943652Z Collecting performance related data for node ip-10-0-131-88.ec2.internal [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:55.080863259Z Collecting performance related data for node ip-10-0-130-235.ec2.internal [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:55.239636094Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:55.502410560Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:55.505071873Z Inspecting resource crd/workloadgroups.networking.istio.io [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:55.507518218Z Gathering data for ns/openshift-multus... [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:55.738234461Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:56.011599038Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:56.163099481Z Inspecting resource mutatingwebhookconfiguration.admissionregistration.k8s.io/istio-sidecar-injector-openshift-gateway-openshift-ingress [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:56.461807742Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:57.031148200Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:57.235288630Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:57.399882220Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:57.448066881Z Gathering data for ns/openshift-ovn-kubernetes... [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:57.537288408Z Inspecting resource validatingwebhookconfiguration.admissionregistration.k8s.io/istio-validator-openshift-gateway-openshift-ingress [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:57.767522725Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:57.875693667Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:58.298535417Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:58.553676452Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:58.745955938Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:59.255696588Z Gathering data for ns/openshift-host-network... [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:59.342814291Z Done executing Istio gather script [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:53:59.699791131Z Gathering data for ns/openshift-network-diagnostics... [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:54:00.376449095Z Gathering data for ns/openshift-network-node-identity... [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:54:00.910740330Z Gathering data for ns/openshift-network-operator... [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:54:01.391063023Z Gathering data for ns/openshift-cloud-network-config-controller... [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:54:01.742184656Z Gathering data for ns/openshift-cluster-node-tuning-operator... [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:54:02.316912027Z Gathering data for ns/openshift-apiserver-operator... [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:54:02.677560108Z Gathering data for ns/openshift-apiserver... [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:54:02.882682178Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:54:03.192197493Z Gathering data for ns/openshift-controller-manager-operator... [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:54:03.557870087Z Gathering data for ns/openshift-controller-manager... [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:54:03.934382641Z Gathering data for ns/openshift-cluster-samples-operator... [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:54:04.341064390Z Collecting kubelet logs for node ip-10-0-141-50.ec2.internal [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:54:04.342981589Z Collecting kubelet logs for node ip-10-0-131-88.ec2.internal [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:54:04.345516419Z Collecting kubelet logs for node ip-10-0-130-235.ec2.internal [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:54:04.730471407Z Gathering data for ns/openshift-operator-lifecycle-manager... [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:54:04.790730800Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-wxwvv namespace [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:54:04.797764882Z INFO: Node performance data collection complete. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:54:05.118402537Z Gathering data for ns/openshift-service-ca-operator... [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:54:05.489441146Z Gathering data for ns/openshift-service-ca... [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:54:05.919384826Z Gathering data for ns/openshift-cluster-csi-drivers... [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:54:06.956110761Z Wrote inspect data to must-gather. [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:54:06.956163028Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:54:06.956163028Z skipping gathering secrets/support due to error: secrets "support" not found [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:54:07.116212618Z error: the server doesn't have a resource type "clusters" [must-gather] [must-gather-hwxvc] POD 2026-07-28T11:54:07.295158379Z Caches written to disk [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.304638588Z waiting for gather to complete [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.308399435Z downloading gather output [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.588834865Z receiving incremental file list [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.614647338Z ./ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.61471759Z aggregated-discovery-api.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.615021576Z aggregated-discovery-apis.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.616014836Z event-filter.html [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.619537708Z timestamp [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.61966927Z version [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.626230773Z cluster-scoped-resources/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.626242793Z cluster-scoped-resources/admissionregistration.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.626262904Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.626323265Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/default-network-annotation.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.626454118Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/servicecidrs.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.626610281Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/user-defined-networks-namespace-label.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.626722953Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.626787785Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/default-network-annotation-binding.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.626902127Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/servicecidrs-binding.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.627006469Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/user-defined-networks-namespace-label-binding.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.627106011Z cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.627157982Z cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/multus.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.627272434Z cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/network-node-identity.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.627356686Z cluster-scoped-resources/apiextensions.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.627868357Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.627915118Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/accounts.nim.opendatahub.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.628112622Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/adminnetworkpolicies.policy.networking.k8s.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.628374497Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/adminpolicybasedexternalroutes.k8s.ovn.org.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.628591301Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertingrules.monitoring.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.628773005Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertmanagerconfigs.monitoring.coreos.com.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.632707505Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertmanagers.monitoring.coreos.com.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.634288657Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertrelabelconfigs.monitoring.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.634513181Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apikeyapprovals.devportal.kuadrant.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.634665784Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apikeyrequests.devportal.kuadrant.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.634788097Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apikeys.devportal.kuadrant.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.63491734Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apiproducts.devportal.kuadrant.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.635488581Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apirequestcounts.apiserver.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.635683535Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apiservers.config.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.635842288Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/appliedmanifestworks.work.open-cluster-management.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.63595369Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authconfigs.authorino.kuadrant.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.636780697Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authentications.config.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.637047683Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authorinos.operator.authorino.kuadrant.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.637174075Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authorizationpolicies.security.istio.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.637341509Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authpolicies.kuadrant.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.638671776Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/baselineadminnetworkpolicies.policy.networking.k8s.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.638907101Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/builds.config.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.639093154Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/bundles.trust.cert-manager.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.639239317Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/catalogsources.operators.coreos.com.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.640556224Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/certificaterequests.cert-manager.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.640772928Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/certificates.cert-manager.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.641005013Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/certmanagers.operator.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.641286039Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/challenges.acme.cert-manager.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.641981133Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/cloudcredentials.operator.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.642102735Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/cloudeventsources.eventing.keda.sh.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.642210357Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/cloudprivateipconfigs.cloud.network.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.64232166Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterclaims.cluster.open-cluster-management.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.642426532Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clustercloudeventsources.eventing.keda.sh.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.642544714Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clustercsidrivers.operator.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.642756648Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterimagepolicies.config.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.642937822Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterissuers.cert-manager.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.643766809Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusteroperators.config.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.643901422Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterresourcequotas.quota.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.644028654Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterserviceversions.operators.coreos.com.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.646074926Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterstoragecontainers.serving.kserve.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.646241979Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clustertriggerauthentications.keda.sh.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.646397753Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusteruserdefinednetworks.k8s.ovn.org.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.646688938Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterversions.config.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.646986134Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/configs.imageregistry.operator.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.647399563Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/configs.operator.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.647515725Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/configs.samples.operator.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.647669288Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleclidownloads.console.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.64778375Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleexternalloglinks.console.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.647891343Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolelinks.console.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.648005285Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolenotifications.console.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.648123408Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleplugins.console.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.64826953Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolequickstarts.console.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.648406273Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoles.config.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.648512835Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoles.operator.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.648809201Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolesamples.console.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.648923494Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleyamlsamples.console.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.649039666Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/controllerconfigs.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.649592637Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/credentialsrequests.cloudcredential.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.64973735Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/csisnapshotcontrollers.operator.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.649851532Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/datascienceclusters.datasciencecluster.opendatahub.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.651119788Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/destinationrules.networking.istio.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.651882904Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnses.config.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.651999356Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnses.operator.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.65218702Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnshealthcheckprobes.kuadrant.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.652293892Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnspolicies.kuadrant.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.652459616Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnsrecords.ingress.operator.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.652593338Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnsrecords.kuadrant.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.652778882Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dscinitializations.dscinitialization.opendatahub.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.652913325Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressfirewalls.k8s.ovn.org.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.653031177Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressips.k8s.ovn.org.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.65316398Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressqoses.k8s.ovn.org.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.653303862Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressrouters.network.operator.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.653455676Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressservices.k8s.ovn.org.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.653599028Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/envoyfilters.networking.istio.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.653755512Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/etcds.operator.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.653889765Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/featuregates.config.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.654007817Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gatewayclasses.gateway.networking.k8s.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.65416123Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.gateway.networking.k8s.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.654516917Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.networking.istio.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.654746282Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/grpcroutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.658919887Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/helmchartrepositories.helm.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.659054299Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/httproutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.659875506Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/httpscaledobjects.http.keda.sh.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.659996638Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagecontentpolicies.config.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.660115911Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagecontentsourcepolicies.operator.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.660228603Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagedigestmirrorsets.config.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.660338735Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagepolicies.config.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.660543509Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagepruners.imageregistry.operator.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.660845616Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/images.config.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.660972228Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagetagmirrorsets.config.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.6610727Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencegraphs.serving.kserve.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.661226983Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencemodelrewrites.llm-d.ai.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.661345506Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferenceobjectives.llm-d.ai.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.661456658Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.k8s.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.661623521Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.661746614Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferenceservices.serving.kserve.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.665765875Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/infrastructures.config.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.666190274Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ingresscontrollers.operator.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.666845427Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ingresses.config.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.667046171Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/insightsoperators.operator.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.667183844Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/installplans.operators.coreos.com.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.667326897Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/interceptorroutes.http.keda.sh.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.66746226Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ipamclaims.k8s.cni.cncf.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.667568942Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ippools.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.667739646Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/issuers.cert-manager.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.668559542Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/istiocsrs.operator.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.668883849Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kedacontrollers.keda.sh.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.675293409Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kuadrants.kuadrant.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.675415681Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubeapiservers.operator.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.675593265Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubecontrollermanagers.operator.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.675747968Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubeschedulers.operator.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.675872251Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubestorageversionmigrators.operator.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.675999053Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/leaderworkersetoperators.operator.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.676122005Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/leaderworkersets.leaderworkerset.x-k8s.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.679084016Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/limitadors.limitador.kuadrant.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.682177939Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/llminferenceserviceconfigs.serving.kserve.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.687850414Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/llminferenceservices.serving.kserve.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.693522438Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineconfignodes.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.693698112Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineconfigurations.operator.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.693984728Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineosbuilds.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.694144291Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineosconfigs.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.694346985Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/network-attachment-definitions.k8s.cni.cncf.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.694453048Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/networks.config.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.694640651Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/networks.operator.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.694901606Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/nodes.config.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.695022129Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/nodeslicepools.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.695121431Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/oauths.config.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.695335945Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/oidcpolicies.extensions.kuadrant.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.695486158Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/olmconfigs.operators.coreos.com.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.6955898Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/openshiftapiservers.operator.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.695728703Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/openshiftcontrollermanagers.operator.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.695833545Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorconditions.operators.coreos.com.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.695972428Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorgroups.operators.coreos.com.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.696143952Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorhubs.config.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.696383077Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorpkis.network.operator.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.696658242Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operators.operators.coreos.com.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.696797605Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/orders.acme.cert-manager.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.696963028Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/overlappingrangeipreservations.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.69705111Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/peerauthentications.security.istio.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.697184503Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/pinnedimagesets.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.697289315Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/planpolicies.extensions.kuadrant.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.697425978Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/podmonitors.monitoring.coreos.com.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.697730974Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/podnetworkconnectivitychecks.controlplane.operator.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.697849046Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/probes.monitoring.coreos.com.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.698094951Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/profiles.tuned.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.698205244Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/projecthelmchartrepositories.helm.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.698325696Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/projects.config.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.698424478Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/prometheuses.monitoring.coreos.com.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.700755255Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/prometheusrules.monitoring.coreos.com.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.700900158Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/proxies.config.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.701009321Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/proxyconfigs.networking.istio.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.701108383Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/rangeallocations.security.internal.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.701214414Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ratelimitpolicies.kuadrant.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.701363718Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/referencegrants.gateway.networking.k8s.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.70148209Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/requestauthentications.security.istio.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.701658363Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/rolebindingrestrictions.authorization.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.701781596Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/scaledjobs.keda.sh.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.70345697Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/scaledobjects.keda.sh.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.703627583Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/schedulers.config.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.703761976Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/securitycontextconstraints.security.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.703907709Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/servicecas.operator.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.704022251Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/serviceentries.networking.istio.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.704199385Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/servicemonitors.monitoring.coreos.com.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.704492191Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/servingruntimes.serving.kserve.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.705312408Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/sidecars.networking.istio.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.705600354Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/storages.operator.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.705731346Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/storagestates.migration.k8s.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.705842779Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/storageversionmigrations.migration.k8s.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.705951701Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/subscriptions.operators.coreos.com.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.706623394Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/telemetries.telemetry.istio.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.706805508Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/telemetrypolicies.extensions.kuadrant.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.7069208Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/thanosrulers.monitoring.coreos.com.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.70836674Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/tlspolicies.kuadrant.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.708510123Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/tokenratelimitpolicies.kuadrant.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.708681176Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/trainedmodels.serving.kserve.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.708788748Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/triggerauthentications.keda.sh.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.708955372Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/trustmanagers.operator.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.709236377Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/tuneds.tuned.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.70936083Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/userdefinednetworks.k8s.ovn.org.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.709535263Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/virtualservices.networking.istio.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.709956202Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumepopulators.populator.storage.k8s.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.710063754Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumesnapshotclasses.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.710179576Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumesnapshotcontents.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.713044655Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumesnapshots.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.713230778Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/wasmplugins.extensions.istio.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.713362961Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadentries.networking.istio.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.713502694Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadgroups.networking.istio.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.713656007Z cluster-scoped-resources/apiregistration.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.713937443Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.713976484Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1..yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.714082026Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.acme.cert-manager.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.714161237Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.admissionregistration.k8s.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.714243149Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apiextensions.k8s.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.714326781Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apiserver.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.714483164Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apps.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.714591666Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apps.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.714712819Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.authentication.k8s.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.71479646Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.authorization.k8s.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.714877982Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.authorization.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.714977994Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.autoscaling.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.715057775Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.batch.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.715140747Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.build.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.715230929Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.cert-manager.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.715317881Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.certificates.k8s.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.715405573Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.cloud.network.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.715490034Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.cloudcredential.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.715587886Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.config.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.715684568Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.console.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.71576775Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.coordination.k8s.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.715852631Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.datasciencecluster.opendatahub.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.715938213Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.discovery.k8s.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.716025235Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.dscinitialization.opendatahub.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.716115577Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.events.k8s.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.716198999Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.flowcontrol.apiserver.k8s.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.71628414Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.gateway.networking.k8s.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.716369672Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.image.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.716457294Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.imageregistry.operator.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.716543556Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.inference.networking.k8s.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.716652528Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.ingress.operator.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.716736509Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.k8s.cni.cncf.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.716822011Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.k8s.ovn.org.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.716907933Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.kuadrant.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.716993125Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.leaderworkerset.x-k8s.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.717084467Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.717167138Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.monitoring.coreos.com.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.71726408Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.monitoring.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.717348232Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.network.operator.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.717435414Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.networking.istio.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.717522816Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.networking.k8s.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.717629618Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.nim.opendatahub.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.717716199Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.node.k8s.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.717804331Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.oauth.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.717893613Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.operator.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.717980435Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.operators.coreos.com.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.718066787Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.packages.operators.coreos.com.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.718157838Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.policy.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.718273551Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.project.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.718364173Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.quota.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.718554616Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.rbac.authorization.k8s.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.718665669Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.resource.k8s.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.718756401Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.route.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.718846592Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.samples.operator.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.718935784Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.scheduling.k8s.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.719026336Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.security.internal.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.719118478Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.security.istio.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.71920737Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.security.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.719335472Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.719429024Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.storage.k8s.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.719554447Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.telemetry.istio.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.71973739Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.template.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.719927494Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.tuned.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.720012336Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.user.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.720113028Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.work.open-cluster-management.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.72022574Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.cluster.open-cluster-management.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.720312752Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.controlplane.operator.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.720399554Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.devportal.kuadrant.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.720491626Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.eventing.keda.sh.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.720596928Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.extensions.istio.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.72070857Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.extensions.kuadrant.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.720800942Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.http.keda.sh.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.720877813Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.k8s.cni.cncf.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.720967705Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.keda.sh.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.721052297Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.kuadrant.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.721140869Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.limitador.kuadrant.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.72122634Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.migration.k8s.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.721316442Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.monitoring.coreos.com.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.721412954Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.operator.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.721498106Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.operators.coreos.com.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.721603328Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.policy.networking.k8s.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.72169363Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.serving.kserve.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.721784402Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.telemetry.istio.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.721874904Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.trust.cert-manager.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.721961815Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.722054227Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.72219334Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.llm-d.ai.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.722306402Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.operators.coreos.com.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.722402244Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.serving.kserve.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.722496777Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha3.networking.istio.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.722615959Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.admissionregistration.k8s.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.722709351Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.external.metrics.k8s.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.722816633Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.gateway.networking.k8s.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.722905815Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.helm.openshift.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.722995856Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.http.keda.sh.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.723082518Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.kuadrant.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.72317044Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.metrics.k8s.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.723267792Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.monitoring.coreos.com.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.723366694Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.networking.istio.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.723455486Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.operator.authorino.kuadrant.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.723547918Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.populator.storage.k8s.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.72366056Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.security.istio.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.723748292Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.serving.kserve.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.723838834Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.storage.k8s.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.723928066Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta3.authorino.kuadrant.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.724018477Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v2.autoscaling.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.724106859Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v2.operators.coreos.com.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.72414724Z cluster-scoped-resources/certificates.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.724185061Z cluster-scoped-resources/certificates.k8s.io/certificatesigningrequests/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.724224261Z cluster-scoped-resources/certificates.k8s.io/certificatesigningrequests/system:openshift:openshift-monitoring-dn8n9.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.724360434Z cluster-scoped-resources/certificates.k8s.io/certificatesigningrequests/system:openshift:openshift-monitoring-fjs9v.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.724478947Z cluster-scoped-resources/certificates.k8s.io/certificatesigningrequests/system:openshift:openshift-monitoring-pvn57.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.724822144Z cluster-scoped-resources/config.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.724915575Z cluster-scoped-resources/config.openshift.io/apiservers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.725169271Z cluster-scoped-resources/config.openshift.io/authentications.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.725310174Z cluster-scoped-resources/config.openshift.io/builds.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.725402625Z cluster-scoped-resources/config.openshift.io/clusterimagepolicies.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.725489997Z cluster-scoped-resources/config.openshift.io/clusteroperators.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.725834134Z cluster-scoped-resources/config.openshift.io/clusterversions.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.725943426Z cluster-scoped-resources/config.openshift.io/consoles.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.726023318Z cluster-scoped-resources/config.openshift.io/dnses.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.72611858Z cluster-scoped-resources/config.openshift.io/featuregates.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.726223952Z cluster-scoped-resources/config.openshift.io/imagecontentpolicies.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.726307144Z cluster-scoped-resources/config.openshift.io/imagedigestmirrorsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.726391565Z cluster-scoped-resources/config.openshift.io/imagepolicies.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.726475707Z cluster-scoped-resources/config.openshift.io/images.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.7265949Z cluster-scoped-resources/config.openshift.io/imagetagmirrorsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.726682571Z cluster-scoped-resources/config.openshift.io/infrastructures.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.726772653Z cluster-scoped-resources/config.openshift.io/ingresses.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.726860305Z cluster-scoped-resources/config.openshift.io/networks.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.726955867Z cluster-scoped-resources/config.openshift.io/nodes.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.727039218Z cluster-scoped-resources/config.openshift.io/oauths.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.7271247Z cluster-scoped-resources/config.openshift.io/operatorhubs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.727239143Z cluster-scoped-resources/config.openshift.io/projects.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.727325324Z cluster-scoped-resources/config.openshift.io/proxies.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.727412806Z cluster-scoped-resources/config.openshift.io/schedulers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.727504298Z cluster-scoped-resources/config.openshift.io/clusteroperators/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.727535349Z cluster-scoped-resources/config.openshift.io/clusteroperators/console.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.727663421Z cluster-scoped-resources/config.openshift.io/clusteroperators/csi-snapshot-controller.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.727756903Z cluster-scoped-resources/config.openshift.io/clusteroperators/dns.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.727842405Z cluster-scoped-resources/config.openshift.io/clusteroperators/image-registry.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.727929167Z cluster-scoped-resources/config.openshift.io/clusteroperators/ingress.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.728021778Z cluster-scoped-resources/config.openshift.io/clusteroperators/insights.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.7281134Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-apiserver.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.728267383Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-controller-manager.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.728355495Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-scheduler.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.728447167Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-storage-version-migrator.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.728536729Z cluster-scoped-resources/config.openshift.io/clusteroperators/monitoring.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.728678112Z cluster-scoped-resources/config.openshift.io/clusteroperators/network.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.728804364Z cluster-scoped-resources/config.openshift.io/clusteroperators/node-tuning.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.728888186Z cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-apiserver.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.729006949Z cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-controller-manager.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.729105071Z cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-samples.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.729194292Z cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager-catalog.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.729279664Z cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager-packageserver.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.729357896Z cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.729449378Z cluster-scoped-resources/config.openshift.io/clusteroperators/service-ca.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.729537709Z cluster-scoped-resources/config.openshift.io/clusteroperators/storage.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.729625381Z cluster-scoped-resources/config.openshift.io/clusterversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.729681022Z cluster-scoped-resources/config.openshift.io/clusterversions/version.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.729730883Z cluster-scoped-resources/config.openshift.io/consoles/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.729774584Z cluster-scoped-resources/config.openshift.io/consoles/cluster.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.729855176Z cluster-scoped-resources/config.openshift.io/featuregates/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.729886116Z cluster-scoped-resources/config.openshift.io/featuregates/cluster.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.729980998Z cluster-scoped-resources/config.openshift.io/infrastructures/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.730037989Z cluster-scoped-resources/config.openshift.io/infrastructures/cluster.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.730094051Z cluster-scoped-resources/config.openshift.io/oauths/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.730137341Z cluster-scoped-resources/config.openshift.io/oauths/cluster.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.730190662Z cluster-scoped-resources/config.openshift.io/proxies/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.730231943Z cluster-scoped-resources/config.openshift.io/proxies/cluster.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.730327135Z cluster-scoped-resources/console.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.730334665Z cluster-scoped-resources/console.openshift.io/consoleplugins/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.730371906Z cluster-scoped-resources/console.openshift.io/consoleplugins/kuadrant-console-plugin.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.73055585Z cluster-scoped-resources/console.openshift.io/consoleplugins/monitoring-plugin.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.730759984Z cluster-scoped-resources/console.openshift.io/consoleplugins/networking-console-plugin.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.730893537Z cluster-scoped-resources/core/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.730912097Z cluster-scoped-resources/core/nodes/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.730938628Z cluster-scoped-resources/core/nodes/ip-10-0-130-235.ec2.internal.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.731131102Z cluster-scoped-resources/core/nodes/ip-10-0-131-88.ec2.internal.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.731269194Z cluster-scoped-resources/core/nodes/ip-10-0-141-50.ec2.internal.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.731404307Z cluster-scoped-resources/core/persistentvolumes/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.731452348Z cluster-scoped-resources/core/persistentvolumes/pvc-22e6f62e-463f-4729-96e2-d0f264811ec3.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.731613671Z cluster-scoped-resources/core/persistentvolumes/pvc-f219be45-903a-4603-991c-a0a969f33047.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.731687563Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.731753764Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.731801865Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/catch-all.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.731929428Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/exempt.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.732015859Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/global-default.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.732104081Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/kube-controller-manager.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.732191353Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/kube-scheduler.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.732279645Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/kube-system-service-accounts.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.732465469Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-apiserver-operator.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.732595991Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-apiserver-sar.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.732710653Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-apiserver.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.732827116Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-authentication-operator.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.732921598Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-controller-manager.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.73302127Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-kube-apiserver-operator.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.733115862Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-monitoring-metrics.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.733224974Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-oauth-apiserver-sar.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.733321806Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-oauth-apiserver.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.733414788Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-oauth-server.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.73354287Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/probes.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.733727024Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/service-accounts.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.733900868Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/system-leader-election.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.734106932Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/system-node-high.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.734229975Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/system-nodes.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.734310556Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.734368487Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/catch-all.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.734462569Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/exempt.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.734552261Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/global-default.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.734678704Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/leader-election.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.734776535Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/node-high.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.734883518Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/openshift-control-plane-operators.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.735043121Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/system.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.735184044Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/workload-high.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.735269565Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/workload-low.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.735329737Z cluster-scoped-resources/gateway.networking.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.735338337Z cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.735390298Z cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/openshift-default.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.735458979Z cluster-scoped-resources/imageregistry.operator.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.73546614Z cluster-scoped-resources/imageregistry.operator.openshift.io/configs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.73551647Z cluster-scoped-resources/imageregistry.operator.openshift.io/configs/cluster.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.735637763Z cluster-scoped-resources/imageregistry.operator.openshift.io/imagepruners/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.735661684Z cluster-scoped-resources/imageregistry.operator.openshift.io/imagepruners/cluster.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.735743235Z cluster-scoped-resources/migration.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.735753055Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.735793306Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/console-plugin-storage-version-migration.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.735889908Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/machineconfiguration-controllerconfig-storage-version-migration.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.736030261Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/machineconfiguration-machineconfigpool-storage-version-migration.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.736079422Z cluster-scoped-resources/oauth.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.736092202Z cluster-scoped-resources/oauth.openshift.io/oauthclients/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.736138673Z cluster-scoped-resources/oauth.openshift.io/oauthclients/console.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.736251045Z cluster-scoped-resources/operator.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.736259666Z cluster-scoped-resources/operator.openshift.io/clustercsidrivers/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.736308797Z cluster-scoped-resources/operator.openshift.io/clustercsidrivers/ebs.csi.aws.com.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.736402699Z cluster-scoped-resources/operator.openshift.io/consoles/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.736443349Z cluster-scoped-resources/operator.openshift.io/consoles/cluster.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.736566302Z cluster-scoped-resources/operator.openshift.io/csisnapshotcontrollers/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.736737495Z cluster-scoped-resources/operator.openshift.io/csisnapshotcontrollers/cluster.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.736823447Z cluster-scoped-resources/operator.openshift.io/dnses/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.736904989Z cluster-scoped-resources/operator.openshift.io/dnses/default.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.737023541Z cluster-scoped-resources/operator.openshift.io/insightsoperators/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.737123313Z cluster-scoped-resources/operator.openshift.io/insightsoperators/cluster.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.737352568Z cluster-scoped-resources/operator.openshift.io/kubeapiservers/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.737403839Z cluster-scoped-resources/operator.openshift.io/kubeapiservers/cluster.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.73748981Z cluster-scoped-resources/operator.openshift.io/kubecontrollermanagers/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.737546032Z cluster-scoped-resources/operator.openshift.io/kubecontrollermanagers/cluster.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.737633073Z cluster-scoped-resources/operator.openshift.io/kubeschedulers/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.737675294Z cluster-scoped-resources/operator.openshift.io/kubeschedulers/cluster.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.737744176Z cluster-scoped-resources/operator.openshift.io/kubestorageversionmigrators/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.737791867Z cluster-scoped-resources/operator.openshift.io/kubestorageversionmigrators/cluster.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.737874288Z cluster-scoped-resources/operator.openshift.io/networks/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.737921209Z cluster-scoped-resources/operator.openshift.io/networks/cluster.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.738004641Z cluster-scoped-resources/operator.openshift.io/openshiftapiservers/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.738023072Z cluster-scoped-resources/operator.openshift.io/openshiftapiservers/cluster.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.738124934Z cluster-scoped-resources/operator.openshift.io/openshiftcontrollermanagers/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.738166744Z cluster-scoped-resources/operator.openshift.io/openshiftcontrollermanagers/cluster.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.738246736Z cluster-scoped-resources/operator.openshift.io/servicecas/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.738293877Z cluster-scoped-resources/operator.openshift.io/servicecas/cluster.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.738384869Z cluster-scoped-resources/operator.openshift.io/storages/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.73843355Z cluster-scoped-resources/operator.openshift.io/storages/cluster.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.738596093Z cluster-scoped-resources/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.738609863Z cluster-scoped-resources/operators.coreos.com/olmconfigs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.738675895Z cluster-scoped-resources/operators.coreos.com/olmconfigs/cluster.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.738776657Z cluster-scoped-resources/operators.coreos.com/operators/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.738894559Z cluster-scoped-resources/operators.coreos.com/operators/authorino-operator.kuadrant-system.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.739019222Z cluster-scoped-resources/operators.coreos.com/operators/dns-operator.kuadrant-system.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.739185315Z cluster-scoped-resources/operators.coreos.com/operators/leader-worker-set.openshift-lws-operator.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.739469241Z cluster-scoped-resources/operators.coreos.com/operators/limitador-operator.kuadrant-system.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.739606504Z cluster-scoped-resources/operators.coreos.com/operators/openshift-cert-manager-operator.cert-manager-operator.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.739756127Z cluster-scoped-resources/operators.coreos.com/operators/openshift-custom-metrics-autoscaler-operator.openshift-keda.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.739889319Z cluster-scoped-resources/operators.coreos.com/operators/rhcl-operator.kuadrant-system.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.740005932Z cluster-scoped-resources/rbac.authorization.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.740079273Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.740142654Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/aws-ebs-csi-driver-operator-clusterrolebinding.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.740246486Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/cloud-network-config-controller.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.740342839Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/cluster-node-tuning-operator.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.740508642Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/cluster-storage-operator-role.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.740680365Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/csi-snapshot-controller-operator-clusterrole.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.740793698Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/csi-snapshot-controller-runner-operator.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.74089826Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/metrics-daemon-sa-rolebinding.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.740999822Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-admission-controller-webhook.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.741105494Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-ancillary-tools.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.741230906Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-cluster-readers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.741335129Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-group.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.741438511Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-transient.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.741583544Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-whereabouts.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.741710816Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/network-diagnostics.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.741814938Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/network-node-identity.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.74192036Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-image-registry-pruner.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.742027113Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-iptables-alerter.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.742220527Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.742435521Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-ovn-kubernetes-node-identity-limited.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.742629625Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-ovn-kubernetes-node-kube-rbac-proxy.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.742741287Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/registry-registry-role.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.742861479Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.74291198Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/aws-ebs-csi-driver-operator-clusterrole.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.743035463Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/cloud-network-config-controller.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.743137595Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/cluster-node-tuning-operator.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.743246547Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/console-extensions-reader.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.743402941Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/console-operator.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.743514853Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/console.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.743645675Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/csi-snapshot-controller-operator-clusterrole.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.743750177Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/helm-chartrepos-viewer.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.74385399Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/metrics-daemon-role.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.743956762Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus-admission-controller-webhook.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.744059834Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus-ancillary-tools.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.744165096Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.744271028Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/net-attach-def-project.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.74437286Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/network-diagnostics.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.744476012Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/network-node-identity.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.744598115Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-csi-snapshot-controller-runner.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.744732668Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-iptables-alerter.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.74484729Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-cluster-reader.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.744951002Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.745053534Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-kube-rbac-proxy.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.745180067Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-node-limited.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.745263038Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-udn-editor.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.74534957Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-udn-viewer.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.745526664Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/project-helm-chartrepository-editor.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.745743208Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-admin.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.74586308Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-basic-user.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.745945892Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-storage-admin.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.746032884Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-view.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.746125856Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:registry.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.746213578Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/whereabouts-cni.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.746268189Z cluster-scoped-resources/samples.operator.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.746283599Z cluster-scoped-resources/samples.operator.openshift.io/configs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.74632482Z cluster-scoped-resources/samples.operator.openshift.io/configs/cluster.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.746378131Z cluster-scoped-resources/snapshot.storage.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.746394881Z cluster-scoped-resources/snapshot.storage.k8s.io/volumesnapshotclasses/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.746434892Z cluster-scoped-resources/snapshot.storage.k8s.io/volumesnapshotclasses/csi-aws-vsc.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.746506584Z cluster-scoped-resources/storage.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.746515254Z cluster-scoped-resources/storage.k8s.io/csidrivers/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.746560185Z cluster-scoped-resources/storage.k8s.io/csidrivers/ebs.csi.aws.com.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.746640526Z cluster-scoped-resources/storage.k8s.io/csinodes/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.746679577Z cluster-scoped-resources/storage.k8s.io/csinodes/ip-10-0-130-235.ec2.internal.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.746772309Z cluster-scoped-resources/storage.k8s.io/csinodes/ip-10-0-131-88.ec2.internal.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.7468485Z cluster-scoped-resources/storage.k8s.io/csinodes/ip-10-0-141-50.ec2.internal.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.746911532Z cluster-scoped-resources/storage.k8s.io/storageclasses/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.746925242Z cluster-scoped-resources/storage.k8s.io/storageclasses/gp2-csi.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.747021014Z cluster-scoped-resources/storage.k8s.io/storageclasses/gp3-csi.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.747068465Z cluster-scoped-resources/storage.k8s.io/volumeattachments/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.747113866Z cluster-scoped-resources/storage.k8s.io/volumeattachments/csi-381c9c3d7dd4736bd08fc89624097dc8a411d3eaef78066e938565a7f68d80a4.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.747244528Z cluster-scoped-resources/storage.k8s.io/volumeattachments/csi-8f70dca98acf5b82535b96ad7d8628a568697a1059f39f1d7d80b52c57c81302.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.74734597Z host_service_logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.747388971Z host_service_logs/masters/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.747458963Z host_service_logs/masters/NetworkManager_service.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.747774609Z host_service_logs/masters/crio_service.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.747865651Z host_service_logs/masters/kubelet_service.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.747958133Z host_service_logs/masters/machine-config-daemon-firstboot_service.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.748102726Z host_service_logs/masters/machine-config-daemon-host_service.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.748232548Z host_service_logs/masters/openvswitch_service.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.74832397Z host_service_logs/masters/ostree-finalize-staged_service.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.748413862Z host_service_logs/masters/ovs-configuration_service.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.748499554Z host_service_logs/masters/ovs-vswitchd_service.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.748598326Z host_service_logs/masters/ovsdb-server_service.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.748699628Z host_service_logs/masters/rpm-ostreed_service.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.748750189Z ingress_controllers/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.748762229Z ingress_controllers/default/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.748766509Z ingress_controllers/default/router-default-65b54959b5-rhznz/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.748821471Z ingress_controllers/default/router-default-65b54959b5-rhznz/haproxy.config [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.748946723Z insights-data/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.748999754Z istio/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.749065896Z istio/aggregated-discovery-api.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.749188308Z istio/aggregated-discovery-apis.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.749751619Z istio/event-filter.html [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.749913423Z istio/timestamp [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.750038455Z istio/cluster-scoped-resources/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.750046465Z istio/cluster-scoped-resources/admissionregistration.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.750051165Z istio/cluster-scoped-resources/admissionregistration.k8s.io/mutatingwebhookconfigurations/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.750119647Z istio/cluster-scoped-resources/admissionregistration.k8s.io/mutatingwebhookconfigurations/istio-sidecar-injector-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.750243479Z istio/cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.75029108Z istio/cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/istio-validator-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.750364042Z istio/cluster-scoped-resources/apiextensions.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.750468474Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.750656518Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authorizationpolicies.security.istio.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.750889582Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/destinationrules.networking.istio.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.751876042Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/envoyfilters.networking.istio.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.752095607Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gatewayclasses.gateway.networking.k8s.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.752272601Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.gateway.networking.k8s.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.75276482Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.networking.istio.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.752958075Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/grpcroutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.753371253Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/httproutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.75424017Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.k8s.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.754450545Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.754630238Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/peerauthentications.security.istio.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.754783592Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/proxyconfigs.networking.istio.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.754914084Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/referencegrants.gateway.networking.k8s.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.755074627Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/requestauthentications.security.istio.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.755271561Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/serviceentries.networking.istio.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.755458925Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/sidecars.networking.istio.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.755783432Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/telemetries.telemetry.istio.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.755960395Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/virtualservices.networking.istio.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.756363124Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/wasmplugins.extensions.istio.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.756488746Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadentries.networking.istio.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.75665499Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadgroups.networking.istio.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.756842153Z istio/cluster-scoped-resources/gateway.networking.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.756850293Z istio/cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.756897434Z istio/cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/openshift-default.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.756978846Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.756987796Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.757006217Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/istio-reader-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.757137789Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/istiod-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.757317323Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/istiod-gateway-controller-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.757388564Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.757435175Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/istio-reader-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.757534797Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/istiod-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.757729761Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/istiod-gateway-controller-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.757803683Z istio/namespaces/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.757814113Z istio/namespaces/e2e-test-llm-inference-service-48639af5/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.757818583Z istio/namespaces/e2e-test-llm-inference-service-48639af5/gateway.networking.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.757824223Z istio/namespaces/e2e-test-llm-inference-service-48639af5/gateway.networking.k8s.io/httproutes/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.757877544Z istio/namespaces/e2e-test-llm-inference-service-48639af5/gateway.networking.k8s.io/httproutes/llmisvc-model-pvc-router-manage-e8706282-kserve-route.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.757975446Z istio/namespaces/e2e-test-llm-inference-service-48639af5/inference.networking.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.757986426Z istio/namespaces/e2e-test-llm-inference-service-48639af5/inference.networking.k8s.io/inferencepools/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.758010557Z istio/namespaces/e2e-test-llm-inference-service-48639af5/inference.networking.k8s.io/inferencepools/llmisvc-model-pvc-router-manage-e8706282-inference-pool.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.758057658Z istio/namespaces/e2e-test-llm-inference-service-48639af5/inference.networking.x-k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.758067568Z istio/namespaces/e2e-test-llm-inference-service-48639af5/inference.networking.x-k8s.io/inferencepools/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.758111359Z istio/namespaces/e2e-test-llm-inference-service-48639af5/inference.networking.x-k8s.io/inferencepools/llmisvc-model-pvc-router-manage-e8706282-inference-pool.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.75816788Z istio/namespaces/e2e-test-llm-inference-service-48639af5/networking.istio.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.75817944Z istio/namespaces/e2e-test-llm-inference-service-48639af5/networking.istio.io/destinationrules/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.758230502Z istio/namespaces/e2e-test-llm-inference-service-48639af5/networking.istio.io/destinationrules/llmisvc-model-pvc-router-manage-e8706282-kserve-scheduler.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.758326403Z istio/namespaces/e2e-test-llm-inference-service-48639af5/networking.istio.io/destinationrules/llmisvc-model-pvc-router-manage-e8706282-kserve-shadow-svc.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.758475367Z istio/namespaces/e2e-test-llm-inference-service-48639af5/networking.istio.io/destinationrules/llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.758545418Z istio/namespaces/e2e-test-llm-inference-service-7c3633d1/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.758553608Z istio/namespaces/e2e-test-llm-inference-service-7c3633d1/gateway.networking.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.758561828Z istio/namespaces/e2e-test-llm-inference-service-7c3633d1/gateway.networking.k8s.io/httproutes/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.75863851Z istio/namespaces/e2e-test-llm-inference-service-7c3633d1/gateway.networking.k8s.io/httproutes/llmisvc-model-pvc-router-manage-2577e794-kserve-route.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.758713221Z istio/namespaces/e2e-test-llm-inference-service-7c3633d1/inference.networking.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.758720601Z istio/namespaces/e2e-test-llm-inference-service-7c3633d1/inference.networking.k8s.io/inferencepools/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.758748202Z istio/namespaces/e2e-test-llm-inference-service-7c3633d1/inference.networking.k8s.io/inferencepools/llmisvc-model-pvc-router-manage-2577e794-inference-pool.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.758817733Z istio/namespaces/e2e-test-llm-inference-service-7c3633d1/inference.networking.x-k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.758824734Z istio/namespaces/e2e-test-llm-inference-service-7c3633d1/inference.networking.x-k8s.io/inferencepools/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.758872724Z istio/namespaces/e2e-test-llm-inference-service-7c3633d1/inference.networking.x-k8s.io/inferencepools/llmisvc-model-pvc-router-manage-2577e794-inference-pool.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.758942236Z istio/namespaces/e2e-test-llm-inference-service-7c3633d1/networking.istio.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.758949196Z istio/namespaces/e2e-test-llm-inference-service-7c3633d1/networking.istio.io/destinationrules/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.758989717Z istio/namespaces/e2e-test-llm-inference-service-7c3633d1/networking.istio.io/destinationrules/llmisvc-model-pvc-router-manage-2577e794-kserve-scheduler.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.759081619Z istio/namespaces/e2e-test-llm-inference-service-7c3633d1/networking.istio.io/destinationrules/llmisvc-model-pvc-router-manage-2577e794-kserve-shadow-svc.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.759218462Z istio/namespaces/e2e-test-llm-inference-service-7c3633d1/networking.istio.io/destinationrules/llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.759380655Z istio/namespaces/openshift-ingress/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.759393915Z istio/namespaces/openshift-ingress/openshift-ingress.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.759469936Z istio/namespaces/openshift-ingress/apps.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.759550978Z istio/namespaces/openshift-ingress/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.75965745Z istio/namespaces/openshift-ingress/apps/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.759696111Z istio/namespaces/openshift-ingress/apps/daemonsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.759810634Z istio/namespaces/openshift-ingress/apps/deployments.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.760100429Z istio/namespaces/openshift-ingress/apps/replicasets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.760307253Z istio/namespaces/openshift-ingress/apps/statefulsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.760350074Z istio/namespaces/openshift-ingress/autoscaling/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.760394985Z istio/namespaces/openshift-ingress/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.760472227Z istio/namespaces/openshift-ingress/batch/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.760531318Z istio/namespaces/openshift-ingress/batch/cronjobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.76064904Z istio/namespaces/openshift-ingress/batch/jobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.760685411Z istio/namespaces/openshift-ingress/build.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.760735502Z istio/namespaces/openshift-ingress/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.760818884Z istio/namespaces/openshift-ingress/build.openshift.io/builds.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.760904306Z istio/namespaces/openshift-ingress/core/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.760953857Z istio/namespaces/openshift-ingress/core/configmaps.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.76159292Z istio/namespaces/openshift-ingress/core/endpoints.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.761705612Z istio/namespaces/openshift-ingress/core/events.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.761929577Z istio/namespaces/openshift-ingress/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.762007518Z istio/namespaces/openshift-ingress/core/pods.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.762279393Z istio/namespaces/openshift-ingress/core/replicationcontrollers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.762405926Z istio/namespaces/openshift-ingress/core/secrets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.762670021Z istio/namespaces/openshift-ingress/core/services.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.762748363Z istio/namespaces/openshift-ingress/discovery.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.762794984Z istio/namespaces/openshift-ingress/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.762872356Z istio/namespaces/openshift-ingress/gateway.networking.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.762882766Z istio/namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.762931887Z istio/namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/openshift-ai-inference.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.763016228Z istio/namespaces/openshift-ingress/image.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.763047679Z istio/namespaces/openshift-ingress/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.763149831Z istio/namespaces/openshift-ingress/k8s.ovn.org/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.763184882Z istio/namespaces/openshift-ingress/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.763288554Z istio/namespaces/openshift-ingress/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.763375786Z istio/namespaces/openshift-ingress/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.763453987Z istio/namespaces/openshift-ingress/monitoring.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.763500118Z istio/namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.76359561Z istio/namespaces/openshift-ingress/networking.istio.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.763608611Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.763681112Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/kuadrant-auth-openshift-ai-inference.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.763783964Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/kuadrant-openshift-ai-inference.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.763922547Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/openshift-ai-inference-authn-ssl.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.764037549Z istio/namespaces/openshift-ingress/networking.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.7640956Z istio/namespaces/openshift-ingress/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.764159362Z istio/namespaces/openshift-ingress/pods/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.764173722Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-w6tmc/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.764225003Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-w6tmc/istiod-openshift-gateway-5677ccb49b-w6tmc.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.764355006Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-w6tmc/discovery/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.764365686Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-w6tmc/discovery/discovery/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.764375326Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-w6tmc/discovery/discovery/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.764421957Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-w6tmc/discovery/discovery/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.77492162Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-w6tmc/discovery/discovery/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.774983701Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-w6tmc/discovery/discovery/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.775028852Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.775073443Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.775171805Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf/istio-proxy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.775187546Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf/istio-proxy/istio-proxy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.775200016Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.775242787Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.776657135Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.776740327Z istio/namespaces/openshift-ingress/pods/router-default-65b54959b5-rhznz/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.776788788Z istio/namespaces/openshift-ingress/pods/router-default-65b54959b5-rhznz/router-default-65b54959b5-rhznz.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.7769041Z istio/namespaces/openshift-ingress/pods/router-default-65b54959b5-rhznz/router/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.776914521Z istio/namespaces/openshift-ingress/pods/router-default-65b54959b5-rhznz/router/router/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.776918601Z istio/namespaces/openshift-ingress/pods/router-default-65b54959b5-rhznz/router/router/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.776926271Z istio/namespaces/openshift-ingress/pods/router-default-65b54959b5-rhznz/router/router/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.777045123Z istio/namespaces/openshift-ingress/pods/router-default-65b54959b5-rhznz/router/router/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.777112914Z istio/namespaces/openshift-ingress/pods/router-default-65b54959b5-rhznz/router/router/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.777156365Z istio/namespaces/openshift-ingress/policy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.777170416Z istio/namespaces/openshift-ingress/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.777225957Z istio/namespaces/openshift-ingress/route.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.777268618Z istio/namespaces/openshift-ingress/route.openshift.io/routes.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.777309509Z machine_config_ondisk/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.777319509Z machine_config_termination_logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.777325599Z monitoring/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.777331299Z monitoring/alertmanager/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.777353039Z monitoring/alertmanager/status.json [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.777438951Z monitoring/alertmanager/status.stderr [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.777493352Z monitoring/prometheus/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.777539873Z monitoring/prometheus/alertmanagers.json [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.777650585Z monitoring/prometheus/alertmanagers.stderr [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.777732577Z monitoring/prometheus/rules.json [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.77837545Z monitoring/prometheus/rules.stderr [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.778417541Z monitoring/prometheus/prometheus-k8s-0/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.778477842Z monitoring/prometheus/prometheus-k8s-0/active-targets.json [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.779506133Z monitoring/prometheus/prometheus-k8s-0/active-targets.stderr [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.779537434Z monitoring/prometheus/prometheus-k8s-0/status/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.779602025Z monitoring/prometheus/prometheus-k8s-0/status/runtimeinfo.json [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.779704537Z monitoring/prometheus/prometheus-k8s-0/status/runtimeinfo.stderr [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.779765968Z monitoring/prometheus/prometheus-k8s-0/status/tsdb.json [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.77985779Z monitoring/prometheus/prometheus-k8s-0/status/tsdb.stderr [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.779901071Z monitoring/prometheus/status/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.779960442Z monitoring/prometheus/status/config.json [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.780430752Z monitoring/prometheus/status/config.stderr [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.780496823Z monitoring/prometheus/status/flags.json [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.780592015Z monitoring/prometheus/status/flags.stderr [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.780903071Z namespaces/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.780912531Z namespaces/cert-manager-operator/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.780917502Z namespaces/cert-manager-operator/coordination.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.780925332Z namespaces/cert-manager-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.780972513Z namespaces/cert-manager-operator/coordination.k8s.io/leases/cert-manager-operator-lock.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.781041094Z namespaces/cert-manager-operator/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.781055925Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.781103555Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.781287519Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.781515324Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.781705498Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.781971773Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.782244318Z namespaces/cert-manager-operator/operators.coreos.com/installplans/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.782281949Z namespaces/cert-manager-operator/operators.coreos.com/installplans/install-r7cls.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.782445543Z namespaces/cert-manager-operator/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.782482023Z namespaces/cert-manager-operator/operators.coreos.com/operatorconditions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.782555655Z namespaces/cert-manager-operator/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.782572685Z namespaces/cert-manager-operator/operators.coreos.com/operatorgroups/openshift-cert-manager-operator.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.782673967Z namespaces/cert-manager-operator/operators.coreos.com/subscriptions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.782689568Z namespaces/cert-manager-operator/operators.coreos.com/subscriptions/openshift-cert-manager-operator.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.7827894Z namespaces/cert-manager/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.7828018Z namespaces/cert-manager/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.7828126Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.782845321Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.782999644Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.783177057Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.783333181Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.783545725Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.78380804Z namespaces/default/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.783847741Z namespaces/default/default.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.783910943Z namespaces/default/apps.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.783959724Z namespaces/default/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.784031995Z namespaces/default/apps/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.784062775Z namespaces/default/apps/daemonsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.784154007Z namespaces/default/apps/deployments.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.784228339Z namespaces/default/apps/replicasets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.78430908Z namespaces/default/apps/statefulsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.784366212Z namespaces/default/autoscaling/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.784411643Z namespaces/default/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.784484704Z namespaces/default/batch/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.784528315Z namespaces/default/batch/cronjobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.784630257Z namespaces/default/batch/jobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.784668078Z namespaces/default/build.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.784700859Z namespaces/default/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.78478531Z namespaces/default/build.openshift.io/builds.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.784816491Z namespaces/default/core/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.784864152Z namespaces/default/core/configmaps.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.784972984Z namespaces/default/core/endpoints.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.785056826Z namespaces/default/core/events.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.785223429Z namespaces/default/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.785305761Z namespaces/default/core/pods.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.785382972Z namespaces/default/core/replicationcontrollers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.785510685Z namespaces/default/core/secrets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.785698929Z namespaces/default/core/services.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.78577403Z namespaces/default/discovery.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.785816051Z namespaces/default/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.785887792Z namespaces/default/image.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.785917463Z namespaces/default/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.785965874Z namespaces/default/k8s.ovn.org/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.785995345Z namespaces/default/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.786081236Z namespaces/default/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.786157718Z namespaces/default/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.786222869Z namespaces/default/monitoring.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.78627638Z namespaces/default/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.786334202Z namespaces/default/networking.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.786383283Z namespaces/default/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.786438734Z namespaces/default/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.786451914Z namespaces/default/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.786490505Z namespaces/default/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.786679159Z namespaces/default/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.786863322Z namespaces/default/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.786997315Z namespaces/default/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.787261991Z namespaces/default/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.787470325Z namespaces/default/policy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.787505705Z namespaces/default/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.787598647Z namespaces/default/route.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.787619218Z namespaces/default/route.openshift.io/routes.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.787685669Z namespaces/e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.787696369Z namespaces/e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.78770772Z namespaces/e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.78775486Z namespaces/e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.787903644Z namespaces/e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.788082427Z namespaces/e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.78821904Z namespaces/e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.788429114Z namespaces/e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.788637878Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.788651648Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.788662129Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.788696329Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.788853223Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.789030316Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.78920516Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.789405384Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.789625308Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.789636339Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.789650419Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.78968997Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.789843343Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.790015206Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.790150859Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.790351903Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.790542807Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.790552847Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.790586108Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.790659739Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.790812983Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.791128979Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.791265381Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.791455176Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.79165434Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.79166821Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.79167931Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.791726401Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.791885114Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.792065788Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.792202971Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.792416575Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.79264992Z namespaces/e2e-test-llm-inference-service-48639af5/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.79266229Z namespaces/e2e-test-llm-inference-service-48639af5/monitoring.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.79267Z namespaces/e2e-test-llm-inference-service-48639af5/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.792725411Z namespaces/e2e-test-llm-inference-service-48639af5/monitoring.coreos.com/podmonitors/kserve-llm-isvc-vllm-engine-default.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.792851754Z namespaces/e2e-test-llm-inference-service-48639af5/monitoring.coreos.com/podmonitors/kserve-llm-isvc-vllm-engine.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.792927635Z namespaces/e2e-test-llm-inference-service-48639af5/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.792974446Z namespaces/e2e-test-llm-inference-service-48639af5/monitoring.coreos.com/servicemonitors/kserve-llm-isvc-scheduler-default.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.793085438Z namespaces/e2e-test-llm-inference-service-48639af5/monitoring.coreos.com/servicemonitors/kserve-llm-isvc-scheduler.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.79313928Z namespaces/e2e-test-llm-inference-service-48639af5/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.79315974Z namespaces/e2e-test-llm-inference-service-48639af5/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.793204951Z namespaces/e2e-test-llm-inference-service-48639af5/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.793365144Z namespaces/e2e-test-llm-inference-service-48639af5/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.793540588Z namespaces/e2e-test-llm-inference-service-48639af5/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.793696181Z namespaces/e2e-test-llm-inference-service-48639af5/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.793898115Z namespaces/e2e-test-llm-inference-service-48639af5/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.794084969Z namespaces/e2e-test-llm-inference-service-7c3633d1/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.794094089Z namespaces/e2e-test-llm-inference-service-7c3633d1/monitoring.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.794099249Z namespaces/e2e-test-llm-inference-service-7c3633d1/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.794111129Z namespaces/e2e-test-llm-inference-service-7c3633d1/monitoring.coreos.com/podmonitors/kserve-llm-isvc-vllm-engine-default.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.794211611Z namespaces/e2e-test-llm-inference-service-7c3633d1/monitoring.coreos.com/podmonitors/kserve-llm-isvc-vllm-engine.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.794257192Z namespaces/e2e-test-llm-inference-service-7c3633d1/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.794295693Z namespaces/e2e-test-llm-inference-service-7c3633d1/monitoring.coreos.com/servicemonitors/kserve-llm-isvc-scheduler-default.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.794377345Z namespaces/e2e-test-llm-inference-service-7c3633d1/monitoring.coreos.com/servicemonitors/kserve-llm-isvc-scheduler.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.794426906Z namespaces/e2e-test-llm-inference-service-7c3633d1/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.794442986Z namespaces/e2e-test-llm-inference-service-7c3633d1/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.794499797Z namespaces/e2e-test-llm-inference-service-7c3633d1/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.794704151Z namespaces/e2e-test-llm-inference-service-7c3633d1/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.794881355Z namespaces/e2e-test-llm-inference-service-7c3633d1/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.795014758Z namespaces/e2e-test-llm-inference-service-7c3633d1/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.795221162Z namespaces/e2e-test-llm-inference-service-7c3633d1/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.795401325Z namespaces/kserve-ci-e2e-test/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.795412046Z namespaces/kserve-ci-e2e-test/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.795423216Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.795468867Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.7956377Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.795822654Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.795957047Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.796159461Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.796383405Z namespaces/kserve/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.796428946Z namespaces/kserve/kserve.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.796495848Z namespaces/kserve/apps.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.796543899Z namespaces/kserve/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.796659051Z namespaces/kserve/apps/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.796700482Z namespaces/kserve/apps/daemonsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.796788864Z namespaces/kserve/apps/deployments.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.79710978Z namespaces/kserve/apps/replicasets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.797336255Z namespaces/kserve/apps/statefulsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.797396886Z namespaces/kserve/autoscaling/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.797438287Z namespaces/kserve/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.797498458Z namespaces/kserve/batch/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.79759648Z namespaces/kserve/batch/cronjobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.797701452Z namespaces/kserve/batch/jobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.797786514Z namespaces/kserve/build.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.797834525Z namespaces/kserve/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.797917067Z namespaces/kserve/build.openshift.io/builds.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.797963717Z namespaces/kserve/coordination.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.797970808Z namespaces/kserve/coordination.k8s.io/leases/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.798015559Z namespaces/kserve/coordination.k8s.io/leases/kserve-controller-manager-leader-lock.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.79809513Z namespaces/kserve/coordination.k8s.io/leases/llminferenceservice-kserve-controller-manager.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.798184762Z namespaces/kserve/coordination.k8s.io/leases/odh-model-controller.opendatahub.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.798280284Z namespaces/kserve/core/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.798308144Z namespaces/kserve/core/configmaps.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.79856951Z namespaces/kserve/core/endpoints.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.798696872Z namespaces/kserve/core/events.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.799003869Z namespaces/kserve/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.79908387Z namespaces/kserve/core/pods.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.799403157Z namespaces/kserve/core/replicationcontrollers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.79953838Z namespaces/kserve/core/secrets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.799836066Z namespaces/kserve/core/services.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.799925747Z namespaces/kserve/discovery.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.799960408Z namespaces/kserve/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.80006345Z namespaces/kserve/image.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.800089771Z namespaces/kserve/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.800167742Z namespaces/kserve/k8s.ovn.org/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.800196083Z namespaces/kserve/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.800295425Z namespaces/kserve/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.800373017Z namespaces/kserve/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.800411987Z namespaces/kserve/monitoring.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.800449118Z namespaces/kserve/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.80056695Z namespaces/kserve/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.800637182Z namespaces/kserve/monitoring.coreos.com/servicemonitors/model-serving-api-metrics.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.800723524Z namespaces/kserve/monitoring.coreos.com/servicemonitors/odh-model-controller-metrics-monitor.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.800772865Z namespaces/kserve/networking.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.800819735Z namespaces/kserve/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.800870156Z namespaces/kserve/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.800880197Z namespaces/kserve/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.800931918Z namespaces/kserve/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.801090261Z namespaces/kserve/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.801265035Z namespaces/kserve/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.801402167Z namespaces/kserve/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.801688973Z namespaces/kserve/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.801896447Z namespaces/kserve/pods/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.801910028Z namespaces/kserve/pods/kserve-controller-manager-6968749498-brk56/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.801956029Z namespaces/kserve/pods/kserve-controller-manager-6968749498-brk56/kserve-controller-manager-6968749498-brk56.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.802052671Z namespaces/kserve/pods/kserve-controller-manager-6968749498-brk56/manager/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.802061371Z namespaces/kserve/pods/kserve-controller-manager-6968749498-brk56/manager/manager/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.802067461Z namespaces/kserve/pods/kserve-controller-manager-6968749498-brk56/manager/manager/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.802111112Z namespaces/kserve/pods/kserve-controller-manager-6968749498-brk56/manager/manager/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.802260515Z namespaces/kserve/pods/kserve-controller-manager-6968749498-brk56/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.802360107Z namespaces/kserve/pods/kserve-controller-manager-6968749498-brk56/manager/manager/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.802436808Z namespaces/kserve/pods/llmisvc-controller-manager-6fdfcc6654-ltdmm/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.80250457Z namespaces/kserve/pods/llmisvc-controller-manager-6fdfcc6654-ltdmm/llmisvc-controller-manager-6fdfcc6654-ltdmm.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.802652893Z namespaces/kserve/pods/llmisvc-controller-manager-6fdfcc6654-ltdmm/manager/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.802671343Z namespaces/kserve/pods/llmisvc-controller-manager-6fdfcc6654-ltdmm/manager/manager/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.802677463Z namespaces/kserve/pods/llmisvc-controller-manager-6fdfcc6654-ltdmm/manager/manager/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.802736374Z namespaces/kserve/pods/llmisvc-controller-manager-6fdfcc6654-ltdmm/manager/manager/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.804815287Z namespaces/kserve/pods/llmisvc-controller-manager-6fdfcc6654-ltdmm/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.804882658Z namespaces/kserve/pods/llmisvc-controller-manager-6fdfcc6654-ltdmm/manager/manager/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.804935789Z namespaces/kserve/pods/model-serving-api-7b954ffc95-tr22l/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.804964019Z namespaces/kserve/pods/model-serving-api-7b954ffc95-tr22l/model-serving-api-7b954ffc95-tr22l.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.805069872Z namespaces/kserve/pods/model-serving-api-7b954ffc95-tr22l/server/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.805082742Z namespaces/kserve/pods/model-serving-api-7b954ffc95-tr22l/server/server/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.805094852Z namespaces/kserve/pods/model-serving-api-7b954ffc95-tr22l/server/server/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.805137593Z namespaces/kserve/pods/model-serving-api-7b954ffc95-tr22l/server/server/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.805241985Z namespaces/kserve/pods/model-serving-api-7b954ffc95-tr22l/server/server/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.805309086Z namespaces/kserve/pods/model-serving-api-7b954ffc95-tr22l/server/server/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.805349777Z namespaces/kserve/pods/odh-model-controller-75d9cdbddf-vgnpz/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.805396618Z namespaces/kserve/pods/odh-model-controller-75d9cdbddf-vgnpz/odh-model-controller-75d9cdbddf-vgnpz.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.80548037Z namespaces/kserve/pods/odh-model-controller-75d9cdbddf-vgnpz/manager/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.80549293Z namespaces/kserve/pods/odh-model-controller-75d9cdbddf-vgnpz/manager/manager/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.805522801Z namespaces/kserve/pods/odh-model-controller-75d9cdbddf-vgnpz/manager/manager/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.805535321Z namespaces/kserve/pods/odh-model-controller-75d9cdbddf-vgnpz/manager/manager/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.839267646Z namespaces/kserve/pods/odh-model-controller-75d9cdbddf-vgnpz/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.839337537Z namespaces/kserve/pods/odh-model-controller-75d9cdbddf-vgnpz/manager/manager/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.839361568Z namespaces/kserve/pods/s3-init-vs6wx/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.839417839Z namespaces/kserve/pods/s3-init-vs6wx/s3-init-vs6wx.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.839564062Z namespaces/kserve/pods/s3-init-vs6wx/download-hf-model/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.839603002Z namespaces/kserve/pods/s3-init-vs6wx/download-hf-model/download-hf-model/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.839617793Z namespaces/kserve/pods/s3-init-vs6wx/download-hf-model/download-hf-model/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.839632933Z namespaces/kserve/pods/s3-init-vs6wx/download-hf-model/download-hf-model/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.839728265Z namespaces/kserve/pods/s3-init-vs6wx/download-hf-model/download-hf-model/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.839794186Z namespaces/kserve/pods/s3-init-vs6wx/download-hf-model/download-hf-model/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.839826907Z namespaces/kserve/pods/s3-init-vs6wx/s3-init/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.839836517Z namespaces/kserve/pods/s3-init-vs6wx/s3-init/s3-init/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.839841457Z namespaces/kserve/pods/s3-init-vs6wx/s3-init/s3-init/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.839880658Z namespaces/kserve/pods/s3-init-vs6wx/s3-init/s3-init/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.840146633Z namespaces/kserve/pods/s3-init-vs6wx/s3-init/s3-init/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.840203525Z namespaces/kserve/pods/s3-init-vs6wx/s3-init/s3-init/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.840252275Z namespaces/kserve/pods/seaweedfs-64568bcd49-dk5fj/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.840278266Z namespaces/kserve/pods/seaweedfs-64568bcd49-dk5fj/seaweedfs-64568bcd49-dk5fj.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.840363168Z namespaces/kserve/pods/seaweedfs-64568bcd49-dk5fj/seaweedfs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.840373918Z namespaces/kserve/pods/seaweedfs-64568bcd49-dk5fj/seaweedfs/seaweedfs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.840384588Z namespaces/kserve/pods/seaweedfs-64568bcd49-dk5fj/seaweedfs/seaweedfs/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.840428899Z namespaces/kserve/pods/seaweedfs-64568bcd49-dk5fj/seaweedfs/seaweedfs/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.840664274Z namespaces/kserve/pods/seaweedfs-64568bcd49-dk5fj/seaweedfs/seaweedfs/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.840729965Z namespaces/kserve/pods/seaweedfs-64568bcd49-dk5fj/seaweedfs/seaweedfs/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.840768376Z namespaces/kserve/policy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.840807897Z namespaces/kserve/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.840838798Z namespaces/kserve/route.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.840880998Z namespaces/kserve/route.openshift.io/routes.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.84096906Z namespaces/kuadrant-system/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.841018311Z namespaces/kuadrant-system/kuadrant-system.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.841095033Z namespaces/kuadrant-system/apps.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.841134653Z namespaces/kuadrant-system/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.841200675Z namespaces/kuadrant-system/apps/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.841239415Z namespaces/kuadrant-system/apps/daemonsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.841314917Z namespaces/kuadrant-system/apps/deployments.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.841634443Z namespaces/kuadrant-system/apps/replicasets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.841905069Z namespaces/kuadrant-system/apps/statefulsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.84196105Z namespaces/kuadrant-system/autoscaling/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.842022542Z namespaces/kuadrant-system/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.842087683Z namespaces/kuadrant-system/batch/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.842130394Z namespaces/kuadrant-system/batch/cronjobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.842210845Z namespaces/kuadrant-system/batch/jobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.842260946Z namespaces/kuadrant-system/build.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.842293367Z namespaces/kuadrant-system/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.84241369Z namespaces/kuadrant-system/build.openshift.io/builds.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.842465011Z namespaces/kuadrant-system/coordination.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.842483961Z namespaces/kuadrant-system/coordination.k8s.io/leases/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.842555732Z namespaces/kuadrant-system/coordination.k8s.io/leases/3745a16e.kuadrant.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.842706305Z namespaces/kuadrant-system/coordination.k8s.io/leases/a3f98d6c.kuadrant.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.842826118Z namespaces/kuadrant-system/coordination.k8s.io/leases/aac3a15d.authorino.kuadrant.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.84294438Z namespaces/kuadrant-system/coordination.k8s.io/leases/f139389e.kuadrant.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.843004741Z namespaces/kuadrant-system/core/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.843058843Z namespaces/kuadrant-system/core/configmaps.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.843226076Z namespaces/kuadrant-system/core/endpoints.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.843339548Z namespaces/kuadrant-system/core/events.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.843793827Z namespaces/kuadrant-system/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.843866949Z namespaces/kuadrant-system/core/pods.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.844197626Z namespaces/kuadrant-system/core/replicationcontrollers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.844336628Z namespaces/kuadrant-system/core/secrets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.844600054Z namespaces/kuadrant-system/core/services.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.844719296Z namespaces/kuadrant-system/discovery.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.844743977Z namespaces/kuadrant-system/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.844866089Z namespaces/kuadrant-system/image.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.84489135Z namespaces/kuadrant-system/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.8449295Z namespaces/kuadrant-system/k8s.ovn.org/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.844976691Z namespaces/kuadrant-system/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.845062313Z namespaces/kuadrant-system/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.845134874Z namespaces/kuadrant-system/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.845177235Z namespaces/kuadrant-system/monitoring.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.845206606Z namespaces/kuadrant-system/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.845249607Z namespaces/kuadrant-system/networking.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.845285168Z namespaces/kuadrant-system/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.845354359Z namespaces/kuadrant-system/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.845366199Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.845437721Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.845654675Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.84587775Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.846038813Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.846259137Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.846594664Z namespaces/kuadrant-system/operators.coreos.com/installplans/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.846661905Z namespaces/kuadrant-system/operators.coreos.com/installplans/install-znljx.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.847006872Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.847041843Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.847168126Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.847281988Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.847398431Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.847466992Z namespaces/kuadrant-system/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.847515563Z namespaces/kuadrant-system/operators.coreos.com/operatorgroups/kuadrant.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.847589784Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.847624425Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/authorino-operator-stable-redhat-operators-openshift-marketplace.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.847728917Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/dns-operator-stable-redhat-operators-openshift-marketplace.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.84789231Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/limitador-operator-stable-redhat-operators-openshift-marketplace.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.847978092Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/rhcl-operator.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.848080524Z namespaces/kuadrant-system/pods/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.848091554Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-5xmkm/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.848128935Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-5xmkm/authorino-7cf4765b85-5xmkm.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.848217487Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-5xmkm/authorino/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.848225037Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-5xmkm/authorino/authorino/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.848234868Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-5xmkm/authorino/authorino/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.848291599Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-5xmkm/authorino/authorino/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.86412038Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-5xmkm/authorino/authorino/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.864190681Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-5xmkm/authorino/authorino/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.864213752Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-tlmfp/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.864285283Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-tlmfp/authorino-operator-56fb6f6856-tlmfp.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.864401535Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-tlmfp/manager/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.864410445Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-tlmfp/manager/manager/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.864418366Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-tlmfp/manager/manager/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.864485067Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-tlmfp/manager/manager/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.864788673Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-tlmfp/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.864860315Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-tlmfp/manager/manager/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.864920286Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-8bdq9/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.864956927Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-8bdq9/dns-operator-controller-manager-ccbdf686-8bdq9.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.865079239Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-8bdq9/manager/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.865094399Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-8bdq9/manager/manager/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.8651035Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-8bdq9/manager/manager/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.86512823Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-8bdq9/manager/manager/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.865245342Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-8bdq9/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.865318654Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-8bdq9/manager/manager/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.865382375Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-srd6m/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.865431216Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-srd6m/kuadrant-console-plugin-598f86468b-srd6m.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.865510298Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-srd6m/kuadrant-console-plugin/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.865520618Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-srd6m/kuadrant-console-plugin/kuadrant-console-plugin/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.865530518Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-srd6m/kuadrant-console-plugin/kuadrant-console-plugin/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.86559096Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-srd6m/kuadrant-console-plugin/kuadrant-console-plugin/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.865679311Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-srd6m/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.865753553Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-srd6m/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.865779873Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-2z85v/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.865829854Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-2z85v/kuadrant-operator-controller-manager-66866f5fcc-2z85v.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.866176381Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-2z85v/manager/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.866184682Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-2z85v/manager/manager/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.866188542Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-2z85v/manager/manager/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.866246683Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-2z85v/manager/manager/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.881252387Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-2z85v/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.881310768Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-2z85v/manager/manager/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.881335959Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-2zlsl/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.88139533Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-2zlsl/limitador-limitador-85d9db4d5d-2zlsl.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.881497272Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-2zlsl/limitador/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.881506232Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-2zlsl/limitador/limitador/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.881515853Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-2zlsl/limitador/limitador/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.881528293Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-2zlsl/limitador/limitador/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.881646166Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-2zlsl/limitador/limitador/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.881710787Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-2zlsl/limitador/limitador/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.881771738Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-5grlb/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.881802349Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-5grlb/limitador-operator-controller-manager-774d94476f-5grlb.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.881917531Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-5grlb/manager/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.881926211Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-5grlb/manager/manager/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.881938611Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-5grlb/manager/manager/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.881995413Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-5grlb/manager/manager/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.882198366Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-5grlb/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.882266198Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-5grlb/manager/manager/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.882313489Z namespaces/kuadrant-system/policy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.88236179Z namespaces/kuadrant-system/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.882405251Z namespaces/kuadrant-system/route.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.882440502Z namespaces/kuadrant-system/route.openshift.io/routes.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.882502763Z namespaces/kube-node-lease/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.882513193Z namespaces/kube-node-lease/coordination.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.882517503Z namespaces/kube-node-lease/coordination.k8s.io/leases/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.882591085Z namespaces/kube-node-lease/coordination.k8s.io/leases/ip-10-0-130-235.ec2.internal.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.882689676Z namespaces/kube-node-lease/coordination.k8s.io/leases/ip-10-0-131-88.ec2.internal.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.882767058Z namespaces/kube-node-lease/coordination.k8s.io/leases/ip-10-0-141-50.ec2.internal.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.882815909Z namespaces/kube-node-lease/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.882825169Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.88288086Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.883080804Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.883302069Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.883447972Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.883710737Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.886485424Z namespaces/kube-public/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.886501824Z namespaces/kube-public/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.886515714Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.886590896Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.88680835Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.887027775Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.887174138Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.887386752Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.887642757Z namespaces/kube-system/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.887698468Z namespaces/kube-system/kube-system.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.887873472Z namespaces/kube-system/apps.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.887918513Z namespaces/kube-system/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.887987554Z namespaces/kube-system/apps/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.888016145Z namespaces/kube-system/apps/daemonsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.888167078Z namespaces/kube-system/apps/deployments.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.888248599Z namespaces/kube-system/apps/replicasets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.888350681Z namespaces/kube-system/apps/statefulsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.888426653Z namespaces/kube-system/autoscaling/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.888486204Z namespaces/kube-system/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.888601467Z namespaces/kube-system/batch/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.888651037Z namespaces/kube-system/batch/cronjobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.88875946Z namespaces/kube-system/batch/jobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.888823121Z namespaces/kube-system/build.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.888860622Z namespaces/kube-system/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.888981304Z namespaces/kube-system/build.openshift.io/builds.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.889030205Z namespaces/kube-system/coordination.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.889036545Z namespaces/kube-system/coordination.k8s.io/leases/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.889097906Z namespaces/kube-system/coordination.k8s.io/leases/apiserver-lzerwbdivghn6ozcxm6emdcyva.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.889215519Z namespaces/kube-system/coordination.k8s.io/leases/cert-manager-cainjector-leader-election.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.889329741Z namespaces/kube-system/coordination.k8s.io/leases/cert-manager-controller.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.889443244Z namespaces/kube-system/coordination.k8s.io/leases/kube-controller-manager.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.889652098Z namespaces/kube-system/coordination.k8s.io/leases/kube-scheduler.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.8897476Z namespaces/kube-system/core/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.889802981Z namespaces/kube-system/core/configmaps.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.890015775Z namespaces/kube-system/core/endpoints.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.890132507Z namespaces/kube-system/core/events.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.890491225Z namespaces/kube-system/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.890627648Z namespaces/kube-system/core/pods.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.890991745Z namespaces/kube-system/core/replicationcontrollers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.891180129Z namespaces/kube-system/core/secrets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.891633228Z namespaces/kube-system/core/services.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.891712139Z namespaces/kube-system/discovery.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.89175263Z namespaces/kube-system/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.891852442Z namespaces/kube-system/image.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.891888283Z namespaces/kube-system/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.891984815Z namespaces/kube-system/k8s.ovn.org/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.892023316Z namespaces/kube-system/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.892144258Z namespaces/kube-system/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.892250021Z namespaces/kube-system/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.892327502Z namespaces/kube-system/monitoring.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.892388933Z namespaces/kube-system/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.892449405Z namespaces/kube-system/networking.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.892507546Z namespaces/kube-system/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.892585647Z namespaces/kube-system/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.892600697Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.892662629Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.892843022Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.893062807Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.893390834Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.893654619Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.893934985Z namespaces/kube-system/pods/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.893944905Z namespaces/kube-system/pods/global-pull-secret-syncer-bffsx/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.894009036Z namespaces/kube-system/pods/global-pull-secret-syncer-bffsx/global-pull-secret-syncer-bffsx.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.894113818Z namespaces/kube-system/pods/global-pull-secret-syncer-bffsx/global-pull-secret-syncer/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.894121008Z namespaces/kube-system/pods/global-pull-secret-syncer-bffsx/global-pull-secret-syncer/global-pull-secret-syncer/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.894125088Z namespaces/kube-system/pods/global-pull-secret-syncer-bffsx/global-pull-secret-syncer/global-pull-secret-syncer/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.89418482Z namespaces/kube-system/pods/global-pull-secret-syncer-bffsx/global-pull-secret-syncer/global-pull-secret-syncer/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.894755651Z namespaces/kube-system/pods/global-pull-secret-syncer-bffsx/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.894840353Z namespaces/kube-system/pods/global-pull-secret-syncer-bffsx/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.894923815Z namespaces/kube-system/pods/global-pull-secret-syncer-jhkxw/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.894979696Z namespaces/kube-system/pods/global-pull-secret-syncer-jhkxw/global-pull-secret-syncer-jhkxw.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.895078778Z namespaces/kube-system/pods/global-pull-secret-syncer-jhkxw/global-pull-secret-syncer/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.895087528Z namespaces/kube-system/pods/global-pull-secret-syncer-jhkxw/global-pull-secret-syncer/global-pull-secret-syncer/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.895092328Z namespaces/kube-system/pods/global-pull-secret-syncer-jhkxw/global-pull-secret-syncer/global-pull-secret-syncer/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.895139309Z namespaces/kube-system/pods/global-pull-secret-syncer-jhkxw/global-pull-secret-syncer/global-pull-secret-syncer/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.895705831Z namespaces/kube-system/pods/global-pull-secret-syncer-jhkxw/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.895794762Z namespaces/kube-system/pods/global-pull-secret-syncer-jhkxw/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.895846923Z namespaces/kube-system/pods/global-pull-secret-syncer-nz956/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.895902325Z namespaces/kube-system/pods/global-pull-secret-syncer-nz956/global-pull-secret-syncer-nz956.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.895982996Z namespaces/kube-system/pods/global-pull-secret-syncer-nz956/global-pull-secret-syncer/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.895991997Z namespaces/kube-system/pods/global-pull-secret-syncer-nz956/global-pull-secret-syncer/global-pull-secret-syncer/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.895998766Z namespaces/kube-system/pods/global-pull-secret-syncer-nz956/global-pull-secret-syncer/global-pull-secret-syncer/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.896049178Z namespaces/kube-system/pods/global-pull-secret-syncer-nz956/global-pull-secret-syncer/global-pull-secret-syncer/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.896548908Z namespaces/kube-system/pods/global-pull-secret-syncer-nz956/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.89666146Z namespaces/kube-system/pods/global-pull-secret-syncer-nz956/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.896701461Z namespaces/kube-system/pods/konnectivity-agent-2w2gl/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.896756782Z namespaces/kube-system/pods/konnectivity-agent-2w2gl/konnectivity-agent-2w2gl.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.896859694Z namespaces/kube-system/pods/konnectivity-agent-2w2gl/konnectivity-agent/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.896869784Z namespaces/kube-system/pods/konnectivity-agent-2w2gl/konnectivity-agent/konnectivity-agent/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.896881484Z namespaces/kube-system/pods/konnectivity-agent-2w2gl/konnectivity-agent/konnectivity-agent/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.896940286Z namespaces/kube-system/pods/konnectivity-agent-2w2gl/konnectivity-agent/konnectivity-agent/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.898558418Z namespaces/kube-system/pods/konnectivity-agent-2w2gl/konnectivity-agent/konnectivity-agent/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.898667471Z namespaces/kube-system/pods/konnectivity-agent-2w2gl/konnectivity-agent/konnectivity-agent/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.898729552Z namespaces/kube-system/pods/konnectivity-agent-8bh2v/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.898771533Z namespaces/kube-system/pods/konnectivity-agent-8bh2v/konnectivity-agent-8bh2v.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.898884385Z namespaces/kube-system/pods/konnectivity-agent-8bh2v/konnectivity-agent/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.898891805Z namespaces/kube-system/pods/konnectivity-agent-8bh2v/konnectivity-agent/konnectivity-agent/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.898898185Z namespaces/kube-system/pods/konnectivity-agent-8bh2v/konnectivity-agent/konnectivity-agent/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.898955857Z namespaces/kube-system/pods/konnectivity-agent-8bh2v/konnectivity-agent/konnectivity-agent/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.900531798Z namespaces/kube-system/pods/konnectivity-agent-8bh2v/konnectivity-agent/konnectivity-agent/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.900647161Z namespaces/kube-system/pods/konnectivity-agent-8bh2v/konnectivity-agent/konnectivity-agent/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.900716132Z namespaces/kube-system/pods/konnectivity-agent-v7gsk/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.900774173Z namespaces/kube-system/pods/konnectivity-agent-v7gsk/konnectivity-agent-v7gsk.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.900859095Z namespaces/kube-system/pods/konnectivity-agent-v7gsk/konnectivity-agent/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.900866325Z namespaces/kube-system/pods/konnectivity-agent-v7gsk/konnectivity-agent/konnectivity-agent/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.900870185Z namespaces/kube-system/pods/konnectivity-agent-v7gsk/konnectivity-agent/konnectivity-agent/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.900924637Z namespaces/kube-system/pods/konnectivity-agent-v7gsk/konnectivity-agent/konnectivity-agent/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.902524279Z namespaces/kube-system/pods/konnectivity-agent-v7gsk/konnectivity-agent/konnectivity-agent/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.902649641Z namespaces/kube-system/pods/konnectivity-agent-v7gsk/konnectivity-agent/konnectivity-agent/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.902682352Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-130-235.ec2.internal/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.902744543Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-130-235.ec2.internal/kube-apiserver-proxy-ip-10-0-130-235.ec2.internal.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.902853096Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-130-235.ec2.internal/haproxy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.902863396Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-130-235.ec2.internal/haproxy/haproxy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.902875826Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-130-235.ec2.internal/haproxy/haproxy/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.902926657Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-130-235.ec2.internal/haproxy/haproxy/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.903478848Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-130-235.ec2.internal/haproxy/haproxy/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.90356566Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-130-235.ec2.internal/haproxy/haproxy/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.903653872Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-88.ec2.internal/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.903697473Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-88.ec2.internal/kube-apiserver-proxy-ip-10-0-131-88.ec2.internal.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.903795215Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-88.ec2.internal/haproxy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.903806435Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-88.ec2.internal/haproxy/haproxy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.903810465Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-88.ec2.internal/haproxy/haproxy/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.903864036Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-88.ec2.internal/haproxy/haproxy/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.904215583Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-88.ec2.internal/haproxy/haproxy/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.904299725Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-88.ec2.internal/haproxy/haproxy/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.904362056Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-141-50.ec2.internal/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.904415337Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-141-50.ec2.internal/kube-apiserver-proxy-ip-10-0-141-50.ec2.internal.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.904514339Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-141-50.ec2.internal/haproxy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.90452517Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-141-50.ec2.internal/haproxy/haproxy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.90452924Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-141-50.ec2.internal/haproxy/haproxy/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.904620661Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-141-50.ec2.internal/haproxy/haproxy/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.904953748Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-141-50.ec2.internal/haproxy/haproxy/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.90503807Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-141-50.ec2.internal/haproxy/haproxy/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.905081061Z namespaces/kube-system/policy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.905139202Z namespaces/kube-system/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.905196123Z namespaces/kube-system/rbac.authorization.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.905206663Z namespaces/kube-system/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.905253284Z namespaces/kube-system/rbac.authorization.k8s.io/rolebindings/csi-snapshot-controller-operator-authentication-reader.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.905373897Z namespaces/kube-system/rbac.authorization.k8s.io/rolebindings/network-diagnostics.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.905436558Z namespaces/kube-system/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.905479199Z namespaces/kube-system/rbac.authorization.k8s.io/roles/extension-apiserver-authentication-reader.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.90555328Z namespaces/kube-system/route.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.905624292Z namespaces/kube-system/route.openshift.io/routes.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.905699783Z namespaces/observability/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.905707103Z namespaces/observability/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.905713394Z namespaces/observability/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.905784525Z namespaces/observability/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.90601471Z namespaces/observability/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.906288715Z namespaces/observability/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.906464709Z namespaces/observability/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.906864667Z namespaces/observability/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.907109282Z namespaces/open-cluster-management-agent-addon/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.907117712Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.907124542Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/leases/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.907190144Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/leases/cluster-proxy.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.907302966Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/leases/work-manager.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.907373997Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.907387628Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.907456669Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.907658313Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.907872778Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.908035861Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.908296546Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.90851394Z namespaces/open-cluster-management-be080280-9636-44bb-bf42-272b50be1/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.908522251Z namespaces/open-cluster-management-be080280-9636-44bb-bf42-272b50be1/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.908528531Z namespaces/open-cluster-management-be080280-9636-44bb-bf42-272b50be1/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.908622913Z namespaces/open-cluster-management-be080280-9636-44bb-bf42-272b50be1/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.908813186Z namespaces/open-cluster-management-be080280-9636-44bb-bf42-272b50be1/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.909022131Z namespaces/open-cluster-management-be080280-9636-44bb-bf42-272b50be1/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.909192604Z namespaces/open-cluster-management-be080280-9636-44bb-bf42-272b50be1/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.909430949Z namespaces/open-cluster-management-be080280-9636-44bb-bf42-272b50be1/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.909722565Z namespaces/openshift-apiserver-operator/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.909782006Z namespaces/openshift-apiserver-operator/openshift-apiserver-operator.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.909867948Z namespaces/openshift-apiserver-operator/apps.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.909930289Z namespaces/openshift-apiserver-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.910010401Z namespaces/openshift-apiserver-operator/apps/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.910057422Z namespaces/openshift-apiserver-operator/apps/daemonsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.910171324Z namespaces/openshift-apiserver-operator/apps/deployments.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.910271056Z namespaces/openshift-apiserver-operator/apps/replicasets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.910371108Z namespaces/openshift-apiserver-operator/apps/statefulsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.910438259Z namespaces/openshift-apiserver-operator/autoscaling/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.91047889Z namespaces/openshift-apiserver-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.910552302Z namespaces/openshift-apiserver-operator/batch/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.910667854Z namespaces/openshift-apiserver-operator/batch/cronjobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.910773346Z namespaces/openshift-apiserver-operator/batch/jobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.910811557Z namespaces/openshift-apiserver-operator/build.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.910857388Z namespaces/openshift-apiserver-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.91097398Z namespaces/openshift-apiserver-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.911386869Z namespaces/openshift-apiserver-operator/core/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.91143301Z namespaces/openshift-apiserver-operator/core/configmaps.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.918151256Z namespaces/openshift-apiserver-operator/core/endpoints.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.918251438Z namespaces/openshift-apiserver-operator/core/events.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.91835332Z namespaces/openshift-apiserver-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.918459062Z namespaces/openshift-apiserver-operator/core/pods.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.918606795Z namespaces/openshift-apiserver-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.918754908Z namespaces/openshift-apiserver-operator/core/secrets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.918978853Z namespaces/openshift-apiserver-operator/core/services.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.919064545Z namespaces/openshift-apiserver-operator/discovery.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.919125616Z namespaces/openshift-apiserver-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.919199477Z namespaces/openshift-apiserver-operator/image.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.91932356Z namespaces/openshift-apiserver-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.919395771Z namespaces/openshift-apiserver-operator/k8s.ovn.org/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.919455953Z namespaces/openshift-apiserver-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.919559884Z namespaces/openshift-apiserver-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.919735708Z namespaces/openshift-apiserver-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.919804569Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.919865131Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.919954612Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.920007864Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/servicemonitors/openshift-apiserver-operator.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.920078335Z namespaces/openshift-apiserver-operator/networking.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.920132556Z namespaces/openshift-apiserver-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.920213808Z namespaces/openshift-apiserver-operator/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.920228018Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.920268939Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.920670817Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.920897072Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.921065455Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.92130552Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.921522375Z namespaces/openshift-apiserver-operator/policy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.921592096Z namespaces/openshift-apiserver-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.921652917Z namespaces/openshift-apiserver-operator/route.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.921700198Z namespaces/openshift-apiserver-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.921836171Z namespaces/openshift-apiserver/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.921893672Z namespaces/openshift-apiserver/openshift-apiserver.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.921983354Z namespaces/openshift-apiserver/apps.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.922039705Z namespaces/openshift-apiserver/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.922103486Z namespaces/openshift-apiserver/apps/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.922151587Z namespaces/openshift-apiserver/apps/daemonsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.922261019Z namespaces/openshift-apiserver/apps/deployments.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.922361221Z namespaces/openshift-apiserver/apps/replicasets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.922462583Z namespaces/openshift-apiserver/apps/statefulsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.922517644Z namespaces/openshift-apiserver/autoscaling/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.922595586Z namespaces/openshift-apiserver/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.922683648Z namespaces/openshift-apiserver/batch/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.922725549Z namespaces/openshift-apiserver/batch/cronjobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.922844941Z namespaces/openshift-apiserver/batch/jobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.922914993Z namespaces/openshift-apiserver/build.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.922976664Z namespaces/openshift-apiserver/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.923079566Z namespaces/openshift-apiserver/build.openshift.io/builds.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.923180508Z namespaces/openshift-apiserver/core/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.923233749Z namespaces/openshift-apiserver/core/configmaps.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.923360762Z namespaces/openshift-apiserver/core/endpoints.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.923464014Z namespaces/openshift-apiserver/core/events.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.923565706Z namespaces/openshift-apiserver/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.923693979Z namespaces/openshift-apiserver/core/pods.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.92378378Z namespaces/openshift-apiserver/core/replicationcontrollers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.923938313Z namespaces/openshift-apiserver/core/secrets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.924127847Z namespaces/openshift-apiserver/core/services.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.924189998Z namespaces/openshift-apiserver/discovery.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.92424599Z namespaces/openshift-apiserver/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.924319621Z namespaces/openshift-apiserver/image.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.924376152Z namespaces/openshift-apiserver/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.924451504Z namespaces/openshift-apiserver/k8s.ovn.org/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.924500675Z namespaces/openshift-apiserver/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.924631007Z namespaces/openshift-apiserver/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.924733769Z namespaces/openshift-apiserver/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.924797131Z namespaces/openshift-apiserver/monitoring.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.924856552Z namespaces/openshift-apiserver/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.924942234Z namespaces/openshift-apiserver/networking.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.924977414Z namespaces/openshift-apiserver/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.925053476Z namespaces/openshift-apiserver/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.925081336Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.925143588Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.925327021Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.925537856Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.9257377Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.925976585Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.926197499Z namespaces/openshift-apiserver/policy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.92623059Z namespaces/openshift-apiserver/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.926320502Z namespaces/openshift-apiserver/route.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.926353982Z namespaces/openshift-apiserver/route.openshift.io/routes.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.926432134Z namespaces/openshift-authentication-operator/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.926440524Z namespaces/openshift-authentication-operator/monitoring.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.926448354Z namespaces/openshift-authentication-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.926508105Z namespaces/openshift-authentication-operator/monitoring.coreos.com/servicemonitors/authentication-operator.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.926626918Z namespaces/openshift-authentication-operator/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.926643048Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.926703489Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.926887573Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.927123028Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.927305132Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.927548177Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.927818692Z namespaces/openshift-authentication/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.927831932Z namespaces/openshift-authentication/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.927842593Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.927911754Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.928091668Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.928305472Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.928467135Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.928738991Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.928954505Z namespaces/openshift-cloud-controller-manager/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.928964435Z namespaces/openshift-cloud-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.928978646Z namespaces/openshift-cloud-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.929030307Z namespaces/openshift-cloud-controller-manager/coordination.k8s.io/leases/cloud-controller-manager.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.929113458Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.929126649Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.92920282Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.929385284Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.929612639Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.929787362Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.930026447Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.930254761Z namespaces/openshift-cloud-credential-operator/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.930261922Z namespaces/openshift-cloud-credential-operator/coordination.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.930266232Z namespaces/openshift-cloud-credential-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.930325353Z namespaces/openshift-cloud-credential-operator/coordination.k8s.io/leases/cloud-credential-operator-leader.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.930400084Z namespaces/openshift-cloud-credential-operator/monitoring.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.930410245Z namespaces/openshift-cloud-credential-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.930463726Z namespaces/openshift-cloud-credential-operator/monitoring.coreos.com/servicemonitors/cloud-credential-operator.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.930555478Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.930595379Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.93066324Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.930971596Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.93118555Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.931362714Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.931625139Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.931887065Z namespaces/openshift-cloud-network-config-controller/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.931926706Z namespaces/openshift-cloud-network-config-controller/openshift-cloud-network-config-controller.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.932019417Z namespaces/openshift-cloud-network-config-controller/apps.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.932059638Z namespaces/openshift-cloud-network-config-controller/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.93216048Z namespaces/openshift-cloud-network-config-controller/apps/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.932197081Z namespaces/openshift-cloud-network-config-controller/apps/daemonsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.932318303Z namespaces/openshift-cloud-network-config-controller/apps/deployments.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.932420036Z namespaces/openshift-cloud-network-config-controller/apps/replicasets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.932521298Z namespaces/openshift-cloud-network-config-controller/apps/statefulsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.932556658Z namespaces/openshift-cloud-network-config-controller/autoscaling/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.93265203Z namespaces/openshift-cloud-network-config-controller/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.932716862Z namespaces/openshift-cloud-network-config-controller/batch/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.932767852Z namespaces/openshift-cloud-network-config-controller/batch/cronjobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.932868115Z namespaces/openshift-cloud-network-config-controller/batch/jobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.932905145Z namespaces/openshift-cloud-network-config-controller/build.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.932970387Z namespaces/openshift-cloud-network-config-controller/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.933070189Z namespaces/openshift-cloud-network-config-controller/build.openshift.io/builds.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.93312532Z namespaces/openshift-cloud-network-config-controller/coordination.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.93313554Z namespaces/openshift-cloud-network-config-controller/coordination.k8s.io/leases/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.933198551Z namespaces/openshift-cloud-network-config-controller/coordination.k8s.io/leases/cloud-network-config-controller-lock.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.933331164Z namespaces/openshift-cloud-network-config-controller/core/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.933367605Z namespaces/openshift-cloud-network-config-controller/core/configmaps.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.933506807Z namespaces/openshift-cloud-network-config-controller/core/endpoints.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.9336311Z namespaces/openshift-cloud-network-config-controller/core/events.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.933738792Z namespaces/openshift-cloud-network-config-controller/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.933838804Z namespaces/openshift-cloud-network-config-controller/core/pods.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.933942126Z namespaces/openshift-cloud-network-config-controller/core/replicationcontrollers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.93411927Z namespaces/openshift-cloud-network-config-controller/core/secrets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.934255533Z namespaces/openshift-cloud-network-config-controller/core/services.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.934313344Z namespaces/openshift-cloud-network-config-controller/core/serviceaccounts/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.934357425Z namespaces/openshift-cloud-network-config-controller/core/serviceaccounts/cloud-network-config-controller.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.934435546Z namespaces/openshift-cloud-network-config-controller/discovery.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.934490438Z namespaces/openshift-cloud-network-config-controller/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.934586409Z namespaces/openshift-cloud-network-config-controller/image.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.934639481Z namespaces/openshift-cloud-network-config-controller/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.934746783Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.934797274Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.934908876Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.935009628Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.935077239Z namespaces/openshift-cloud-network-config-controller/monitoring.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.935134471Z namespaces/openshift-cloud-network-config-controller/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.935189572Z namespaces/openshift-cloud-network-config-controller/networking.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.935244343Z namespaces/openshift-cloud-network-config-controller/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.935301414Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.935308594Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.935354355Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.935540519Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.935847205Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.936013698Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.936253083Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.936467707Z namespaces/openshift-cloud-network-config-controller/policy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.936501268Z namespaces/openshift-cloud-network-config-controller/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.93659353Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.93660379Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.936650771Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/rolebindings/cloud-network-config-controller-rb.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.936727903Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.936780194Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/roles/cloud-network-config-controller.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.936852015Z namespaces/openshift-cloud-network-config-controller/route.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.936904076Z namespaces/openshift-cloud-network-config-controller/route.openshift.io/routes.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.937030779Z namespaces/openshift-cluster-csi-drivers/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.93708705Z namespaces/openshift-cluster-csi-drivers/openshift-cluster-csi-drivers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.937416257Z namespaces/openshift-cluster-csi-drivers/apps.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.937469448Z namespaces/openshift-cluster-csi-drivers/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.93758676Z namespaces/openshift-cluster-csi-drivers/apps/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.937645811Z namespaces/openshift-cluster-csi-drivers/apps/daemonsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.937807345Z namespaces/openshift-cluster-csi-drivers/apps/deployments.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.937909607Z namespaces/openshift-cluster-csi-drivers/apps/replicasets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.938012849Z namespaces/openshift-cluster-csi-drivers/apps/statefulsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.9380693Z namespaces/openshift-cluster-csi-drivers/autoscaling/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.938127931Z namespaces/openshift-cluster-csi-drivers/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.938195883Z namespaces/openshift-cluster-csi-drivers/batch/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.938257314Z namespaces/openshift-cluster-csi-drivers/batch/cronjobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.938360966Z namespaces/openshift-cluster-csi-drivers/batch/jobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.938432347Z namespaces/openshift-cluster-csi-drivers/build.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.938491339Z namespaces/openshift-cluster-csi-drivers/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.938631671Z namespaces/openshift-cluster-csi-drivers/build.openshift.io/builds.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.938705743Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.938712803Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.938763364Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/ebs-csi-aws-com.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.938879407Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/external-attacher-leader-ebs-csi-aws-com.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.938993339Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/external-resizer-ebs-csi-aws-com.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.939134572Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/external-snapshotter-leader-ebs-csi-aws-com.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.939270544Z namespaces/openshift-cluster-csi-drivers/core/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.939323696Z namespaces/openshift-cluster-csi-drivers/core/configmaps.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.939449988Z namespaces/openshift-cluster-csi-drivers/core/endpoints.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.93955122Z namespaces/openshift-cluster-csi-drivers/core/events.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.939933868Z namespaces/openshift-cluster-csi-drivers/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.94000829Z namespaces/openshift-cluster-csi-drivers/core/pods.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.940222114Z namespaces/openshift-cluster-csi-drivers/core/replicationcontrollers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.940365697Z namespaces/openshift-cluster-csi-drivers/core/secrets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.940465759Z namespaces/openshift-cluster-csi-drivers/core/services.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.94051888Z namespaces/openshift-cluster-csi-drivers/core/serviceaccounts/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.94054727Z namespaces/openshift-cluster-csi-drivers/core/serviceaccounts/aws-ebs-csi-driver-operator.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.940633442Z namespaces/openshift-cluster-csi-drivers/discovery.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.940688473Z namespaces/openshift-cluster-csi-drivers/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.940731374Z namespaces/openshift-cluster-csi-drivers/image.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.940776955Z namespaces/openshift-cluster-csi-drivers/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.940811426Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.940862257Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.940944518Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.94101778Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.941055501Z namespaces/openshift-cluster-csi-drivers/monitoring.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.941084091Z namespaces/openshift-cluster-csi-drivers/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.941140073Z namespaces/openshift-cluster-csi-drivers/networking.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.941181953Z namespaces/openshift-cluster-csi-drivers/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.941261525Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.941271935Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.941318166Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.941472939Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.941688024Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.941836687Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.942031311Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.942233775Z namespaces/openshift-cluster-csi-drivers/pods/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.942241715Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-2h2b9/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.942288266Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-2h2b9/aws-ebs-csi-driver-node-2h2b9.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.942425439Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-2h2b9/csi-driver/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.942433659Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-2h2b9/csi-driver/csi-driver/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.942438089Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-2h2b9/csi-driver/csi-driver/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.94248008Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-2h2b9/csi-driver/csi-driver/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.942594402Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-2h2b9/csi-driver/csi-driver/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.942754405Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-2h2b9/csi-driver/csi-driver/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.942783286Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-2h2b9/csi-liveness-probe/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.942790416Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-2h2b9/csi-liveness-probe/csi-liveness-probe/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.942799816Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-2h2b9/csi-liveness-probe/csi-liveness-probe/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.942853007Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-2h2b9/csi-liveness-probe/csi-liveness-probe/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.942938229Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-2h2b9/csi-liveness-probe/csi-liveness-probe/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.94300483Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-2h2b9/csi-liveness-probe/csi-liveness-probe/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.943048971Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-2h2b9/csi-node-driver-registrar/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.943058531Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-2h2b9/csi-node-driver-registrar/csi-node-driver-registrar/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.943066561Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-2h2b9/csi-node-driver-registrar/csi-node-driver-registrar/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.943112362Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-2h2b9/csi-node-driver-registrar/csi-node-driver-registrar/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.943214535Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-2h2b9/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.943273156Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-2h2b9/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.943315226Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-7sp7b/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.943361378Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-7sp7b/aws-ebs-csi-driver-node-7sp7b.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.943453569Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-7sp7b/csi-driver/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.94346752Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-7sp7b/csi-driver/csi-driver/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.94347359Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-7sp7b/csi-driver/csi-driver/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.943525581Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-7sp7b/csi-driver/csi-driver/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.943656613Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-7sp7b/csi-driver/csi-driver/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.943740875Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-7sp7b/csi-driver/csi-driver/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.943775526Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-7sp7b/csi-liveness-probe/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.943785806Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-7sp7b/csi-liveness-probe/csi-liveness-probe/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.943793976Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-7sp7b/csi-liveness-probe/csi-liveness-probe/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.943834487Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-7sp7b/csi-liveness-probe/csi-liveness-probe/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.943929239Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-7sp7b/csi-liveness-probe/csi-liveness-probe/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.943998471Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-7sp7b/csi-liveness-probe/csi-liveness-probe/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.944029831Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-7sp7b/csi-node-driver-registrar/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.944036241Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-7sp7b/csi-node-driver-registrar/csi-node-driver-registrar/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.944045611Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-7sp7b/csi-node-driver-registrar/csi-node-driver-registrar/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.944092202Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-7sp7b/csi-node-driver-registrar/csi-node-driver-registrar/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.944183124Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-7sp7b/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.944244385Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-7sp7b/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.944409809Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hh49h/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.944444209Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hh49h/aws-ebs-csi-driver-node-hh49h.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.944540061Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hh49h/csi-driver/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.944548882Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hh49h/csi-driver/csi-driver/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.944557572Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hh49h/csi-driver/csi-driver/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.944632553Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hh49h/csi-driver/csi-driver/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.944747066Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hh49h/csi-driver/csi-driver/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.944816067Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hh49h/csi-driver/csi-driver/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.944863858Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hh49h/csi-liveness-probe/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.944872538Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hh49h/csi-liveness-probe/csi-liveness-probe/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.944878938Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hh49h/csi-liveness-probe/csi-liveness-probe/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.944907629Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hh49h/csi-liveness-probe/csi-liveness-probe/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.945000651Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hh49h/csi-liveness-probe/csi-liveness-probe/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.945068792Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hh49h/csi-liveness-probe/csi-liveness-probe/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.945122553Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hh49h/csi-node-driver-registrar/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.945132293Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hh49h/csi-node-driver-registrar/csi-node-driver-registrar/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.945138534Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hh49h/csi-node-driver-registrar/csi-node-driver-registrar/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.945174364Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hh49h/csi-node-driver-registrar/csi-node-driver-registrar/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.945267916Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hh49h/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.945331207Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hh49h/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.945364808Z namespaces/openshift-cluster-csi-drivers/policy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.945410249Z namespaces/openshift-cluster-csi-drivers/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.94547539Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.945484641Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.945518221Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/rolebindings/aws-ebs-csi-driver-operator-rolebinding.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.945594053Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.945653954Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/roles/aws-ebs-csi-driver-operator-role.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.945723735Z namespaces/openshift-cluster-csi-drivers/route.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.945757326Z namespaces/openshift-cluster-csi-drivers/route.openshift.io/routes.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.945822757Z namespaces/openshift-cluster-machine-approver/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.945830377Z namespaces/openshift-cluster-machine-approver/coordination.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.945842538Z namespaces/openshift-cluster-machine-approver/coordination.k8s.io/leases/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.945882589Z namespaces/openshift-cluster-machine-approver/coordination.k8s.io/leases/cluster-machine-approver-leader.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.94594592Z namespaces/openshift-cluster-machine-approver/monitoring.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.94595369Z namespaces/openshift-cluster-machine-approver/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.945992651Z namespaces/openshift-cluster-machine-approver/monitoring.coreos.com/servicemonitors/cluster-machine-approver.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.946047162Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.946058042Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.946118123Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.946411489Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.946631024Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.946791377Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.946986911Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.947215786Z namespaces/openshift-cluster-node-tuning-operator/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.947250926Z namespaces/openshift-cluster-node-tuning-operator/openshift-cluster-node-tuning-operator.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.947379509Z namespaces/openshift-cluster-node-tuning-operator/apps.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.94741429Z namespaces/openshift-cluster-node-tuning-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.947494941Z namespaces/openshift-cluster-node-tuning-operator/apps/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.947529522Z namespaces/openshift-cluster-node-tuning-operator/apps/daemonsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.947700215Z namespaces/openshift-cluster-node-tuning-operator/apps/deployments.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.947783727Z namespaces/openshift-cluster-node-tuning-operator/apps/replicasets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.947856129Z namespaces/openshift-cluster-node-tuning-operator/apps/statefulsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.947961241Z namespaces/openshift-cluster-node-tuning-operator/apps/daemonsets/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.947984061Z namespaces/openshift-cluster-node-tuning-operator/apps/daemonsets/tuned.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.948079903Z namespaces/openshift-cluster-node-tuning-operator/autoscaling/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.948108934Z namespaces/openshift-cluster-node-tuning-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.948164325Z namespaces/openshift-cluster-node-tuning-operator/batch/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.948220136Z namespaces/openshift-cluster-node-tuning-operator/batch/cronjobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.948305508Z namespaces/openshift-cluster-node-tuning-operator/batch/jobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.948339699Z namespaces/openshift-cluster-node-tuning-operator/build.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.948369059Z namespaces/openshift-cluster-node-tuning-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.948462851Z namespaces/openshift-cluster-node-tuning-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.948491622Z namespaces/openshift-cluster-node-tuning-operator/coordination.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.948505112Z namespaces/openshift-cluster-node-tuning-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.948556323Z namespaces/openshift-cluster-node-tuning-operator/coordination.k8s.io/leases/node-tuning-operator-lock.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.948693916Z namespaces/openshift-cluster-node-tuning-operator/core/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.948734687Z namespaces/openshift-cluster-node-tuning-operator/core/configmaps.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.949704996Z namespaces/openshift-cluster-node-tuning-operator/core/endpoints.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.949806318Z namespaces/openshift-cluster-node-tuning-operator/core/events.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.949960711Z namespaces/openshift-cluster-node-tuning-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.950038763Z namespaces/openshift-cluster-node-tuning-operator/core/pods.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.950236057Z namespaces/openshift-cluster-node-tuning-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.95038495Z namespaces/openshift-cluster-node-tuning-operator/core/secrets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.950525953Z namespaces/openshift-cluster-node-tuning-operator/core/services.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.950702736Z namespaces/openshift-cluster-node-tuning-operator/core/serviceaccounts/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.950727807Z namespaces/openshift-cluster-node-tuning-operator/core/serviceaccounts/cluster-node-tuning-operator.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.950789168Z namespaces/openshift-cluster-node-tuning-operator/discovery.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.95086391Z namespaces/openshift-cluster-node-tuning-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.950929291Z namespaces/openshift-cluster-node-tuning-operator/image.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.950961002Z namespaces/openshift-cluster-node-tuning-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.951034813Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.951064074Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.951156906Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.951230837Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.951296968Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.951330679Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.951406181Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.951474782Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/prometheusrules/node-tuning-operator.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.951555724Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.951610275Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/servicemonitors/node-tuning-operator.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.951694427Z namespaces/openshift-cluster-node-tuning-operator/networking.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.951728097Z namespaces/openshift-cluster-node-tuning-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.951771678Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.951797689Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.95184999Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.952045404Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.952241558Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.952387351Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.952587525Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.952891491Z namespaces/openshift-cluster-node-tuning-operator/pods/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.952899521Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-fmf4x/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.952949942Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-fmf4x/tuned-fmf4x.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.953056804Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-fmf4x/tuned/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.953066564Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-fmf4x/tuned/tuned/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.953077865Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-fmf4x/tuned/tuned/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.953121745Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-fmf4x/tuned/tuned/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.953236558Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-fmf4x/tuned/tuned/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.953301199Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-fmf4x/tuned/tuned/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.95336629Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-lcrsz/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.953385311Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-lcrsz/tuned-lcrsz.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.953484483Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-lcrsz/tuned/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.953493963Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-lcrsz/tuned/tuned/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.953502663Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-lcrsz/tuned/tuned/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.953512143Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-lcrsz/tuned/tuned/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.953681847Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-lcrsz/tuned/tuned/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.953757798Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-lcrsz/tuned/tuned/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.953796599Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-msf7n/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.95386372Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-msf7n/tuned-msf7n.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.953958282Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-msf7n/tuned/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.953967483Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-msf7n/tuned/tuned/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.953973693Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-msf7n/tuned/tuned/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.954009163Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-msf7n/tuned/tuned/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.954125896Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-msf7n/tuned/tuned/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.954189557Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-msf7n/tuned/tuned/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.954224058Z namespaces/openshift-cluster-node-tuning-operator/policy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.954258969Z namespaces/openshift-cluster-node-tuning-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.954290599Z namespaces/openshift-cluster-node-tuning-operator/route.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.95432736Z namespaces/openshift-cluster-node-tuning-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.954374621Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.954383991Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.954443152Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ip-10-0-130-235.ec2.internal.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.954544934Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ip-10-0-131-88.ec2.internal.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.954681077Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ip-10-0-141-50.ec2.internal.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.954737768Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/tuneds/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.954775849Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/tuneds/default.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.954867661Z namespaces/openshift-cluster-samples-operator/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.954907142Z namespaces/openshift-cluster-samples-operator/openshift-cluster-samples-operator.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.954971513Z namespaces/openshift-cluster-samples-operator/apps.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.955020414Z namespaces/openshift-cluster-samples-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.955089316Z namespaces/openshift-cluster-samples-operator/apps/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.955116166Z namespaces/openshift-cluster-samples-operator/apps/daemonsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.955207378Z namespaces/openshift-cluster-samples-operator/apps/deployments.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.95531604Z namespaces/openshift-cluster-samples-operator/apps/replicasets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.955414652Z namespaces/openshift-cluster-samples-operator/apps/statefulsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.955470773Z namespaces/openshift-cluster-samples-operator/autoscaling/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.955498724Z namespaces/openshift-cluster-samples-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.955557175Z namespaces/openshift-cluster-samples-operator/batch/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.955705518Z namespaces/openshift-cluster-samples-operator/batch/cronjobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.95578943Z namespaces/openshift-cluster-samples-operator/batch/jobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.955849561Z namespaces/openshift-cluster-samples-operator/build.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.955878512Z namespaces/openshift-cluster-samples-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.955967933Z namespaces/openshift-cluster-samples-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.956039195Z namespaces/openshift-cluster-samples-operator/core/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.956070155Z namespaces/openshift-cluster-samples-operator/core/configmaps.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.956262709Z namespaces/openshift-cluster-samples-operator/core/endpoints.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.956342881Z namespaces/openshift-cluster-samples-operator/core/events.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.956477184Z namespaces/openshift-cluster-samples-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.956552755Z namespaces/openshift-cluster-samples-operator/core/pods.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.956722909Z namespaces/openshift-cluster-samples-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.956846131Z namespaces/openshift-cluster-samples-operator/core/secrets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.957019015Z namespaces/openshift-cluster-samples-operator/core/services.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.957072836Z namespaces/openshift-cluster-samples-operator/discovery.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.957099596Z namespaces/openshift-cluster-samples-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.957456264Z namespaces/openshift-cluster-samples-operator/image.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.957502805Z namespaces/openshift-cluster-samples-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.957594506Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.957655297Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.95775037Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.957940443Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.957982974Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.957999804Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.958073056Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.958124687Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/prometheusrules/samples-operator-alerts.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.958203899Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.958238739Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/servicemonitors/cluster-samples-operator.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.95829215Z namespaces/openshift-cluster-samples-operator/networking.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.958321091Z namespaces/openshift-cluster-samples-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.958376952Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.958384672Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.958420413Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.958599487Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.958803661Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.958931603Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.959130847Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.959328001Z namespaces/openshift-cluster-samples-operator/pods/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.959336511Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-vjpcz/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.959371612Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-vjpcz/cluster-samples-operator-6748489656-vjpcz.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.959497075Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-vjpcz/cluster-samples-operator-watch/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.959504375Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-vjpcz/cluster-samples-operator-watch/cluster-samples-operator-watch/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.959509035Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-vjpcz/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.959554026Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-vjpcz/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.95977324Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-vjpcz/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.959844762Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-vjpcz/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.959881193Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-vjpcz/cluster-samples-operator/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.959889933Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-vjpcz/cluster-samples-operator/cluster-samples-operator/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.959898733Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-vjpcz/cluster-samples-operator/cluster-samples-operator/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.959955244Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-vjpcz/cluster-samples-operator/cluster-samples-operator/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.960688979Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-vjpcz/cluster-samples-operator/cluster-samples-operator/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.96075932Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-vjpcz/cluster-samples-operator/cluster-samples-operator/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.960803761Z namespaces/openshift-cluster-samples-operator/policy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.960846262Z namespaces/openshift-cluster-samples-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.960907923Z namespaces/openshift-cluster-samples-operator/route.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.960942514Z namespaces/openshift-cluster-samples-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.961040176Z namespaces/openshift-cluster-storage-operator/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.961073397Z namespaces/openshift-cluster-storage-operator/openshift-cluster-storage-operator.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.961151068Z namespaces/openshift-cluster-storage-operator/apps.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.961173039Z namespaces/openshift-cluster-storage-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.96123997Z namespaces/openshift-cluster-storage-operator/apps/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.961270741Z namespaces/openshift-cluster-storage-operator/apps/daemonsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.961363443Z namespaces/openshift-cluster-storage-operator/apps/deployments.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.961461895Z namespaces/openshift-cluster-storage-operator/apps/replicasets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.961561157Z namespaces/openshift-cluster-storage-operator/apps/statefulsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.961644538Z namespaces/openshift-cluster-storage-operator/autoscaling/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.96169511Z namespaces/openshift-cluster-storage-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.961745591Z namespaces/openshift-cluster-storage-operator/batch/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.961802132Z namespaces/openshift-cluster-storage-operator/batch/cronjobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.961879523Z namespaces/openshift-cluster-storage-operator/batch/jobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.961914134Z namespaces/openshift-cluster-storage-operator/build.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.961953075Z namespaces/openshift-cluster-storage-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.962035356Z namespaces/openshift-cluster-storage-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.962089887Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.962099227Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.962158299Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/leases/data-source-validator-leader.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.962241361Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/leases/snapshot-controller-leader.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.962335012Z namespaces/openshift-cluster-storage-operator/core/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.962364803Z namespaces/openshift-cluster-storage-operator/core/configmaps.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.962487175Z namespaces/openshift-cluster-storage-operator/core/endpoints.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.962592548Z namespaces/openshift-cluster-storage-operator/core/events.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.962918064Z namespaces/openshift-cluster-storage-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.962993246Z namespaces/openshift-cluster-storage-operator/core/pods.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.963159359Z namespaces/openshift-cluster-storage-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.963307182Z namespaces/openshift-cluster-storage-operator/core/secrets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.963449005Z namespaces/openshift-cluster-storage-operator/core/services.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.963510996Z namespaces/openshift-cluster-storage-operator/core/serviceaccounts/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.963545497Z namespaces/openshift-cluster-storage-operator/core/serviceaccounts/csi-snapshot-controller-operator.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.963644279Z namespaces/openshift-cluster-storage-operator/discovery.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.96369785Z namespaces/openshift-cluster-storage-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.963769492Z namespaces/openshift-cluster-storage-operator/image.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.963795902Z namespaces/openshift-cluster-storage-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.963878174Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.963913415Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.964001436Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.964079398Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.964138609Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.96418348Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.964244271Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.964273502Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/prometheusrules/prometheus.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.964373644Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.964416185Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/servicemonitors/cluster-storage-operator.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.964470136Z namespaces/openshift-cluster-storage-operator/networking.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.964503236Z namespaces/openshift-cluster-storage-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.964606689Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.96465627Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.96470758Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.964865934Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.965041917Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.96517662Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.965378784Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.965589278Z namespaces/openshift-cluster-storage-operator/pods/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.965600429Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-nljwq/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.965639199Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-nljwq/volume-data-source-validator-549c6d54cf-nljwq.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.965729181Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-nljwq/volume-data-source-validator/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.965736692Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-nljwq/volume-data-source-validator/volume-data-source-validator/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.965741072Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-nljwq/volume-data-source-validator/volume-data-source-validator/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.965778302Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-nljwq/volume-data-source-validator/volume-data-source-validator/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.965875694Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-nljwq/volume-data-source-validator/volume-data-source-validator/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.965948646Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-nljwq/volume-data-source-validator/volume-data-source-validator/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.965963326Z namespaces/openshift-cluster-storage-operator/policy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.966015417Z namespaces/openshift-cluster-storage-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.966078979Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.966087409Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.966098899Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/rolebindings/csi-snapshot-controller-operator-role.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.96617669Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.966221341Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/roles/csi-snapshot-controller-operator-role.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.966290083Z namespaces/openshift-cluster-storage-operator/route.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.966325364Z namespaces/openshift-cluster-storage-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.966411505Z namespaces/openshift-cluster-version/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.966444786Z namespaces/openshift-cluster-version/openshift-cluster-version.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.966517397Z namespaces/openshift-cluster-version/apps.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.966557308Z namespaces/openshift-cluster-version/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.96664551Z namespaces/openshift-cluster-version/apps/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.966698951Z namespaces/openshift-cluster-version/apps/daemonsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.966784373Z namespaces/openshift-cluster-version/apps/deployments.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.966860104Z namespaces/openshift-cluster-version/apps/replicasets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.966942146Z namespaces/openshift-cluster-version/apps/statefulsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.966991617Z namespaces/openshift-cluster-version/autoscaling/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.967031808Z namespaces/openshift-cluster-version/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.967071808Z namespaces/openshift-cluster-version/batch/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.96712492Z namespaces/openshift-cluster-version/batch/cronjobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.967201921Z namespaces/openshift-cluster-version/batch/jobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.967263062Z namespaces/openshift-cluster-version/build.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.967295193Z namespaces/openshift-cluster-version/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.967381975Z namespaces/openshift-cluster-version/build.openshift.io/builds.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.967448356Z namespaces/openshift-cluster-version/coordination.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.967455236Z namespaces/openshift-cluster-version/coordination.k8s.io/leases/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.967507927Z namespaces/openshift-cluster-version/coordination.k8s.io/leases/version.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.967592989Z namespaces/openshift-cluster-version/core/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.967713661Z namespaces/openshift-cluster-version/core/configmaps.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.967808254Z namespaces/openshift-cluster-version/core/endpoints.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.967886585Z namespaces/openshift-cluster-version/core/events.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.968020928Z namespaces/openshift-cluster-version/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.968095509Z namespaces/openshift-cluster-version/core/pods.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.968177961Z namespaces/openshift-cluster-version/core/replicationcontrollers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.968297333Z namespaces/openshift-cluster-version/core/secrets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.968459987Z namespaces/openshift-cluster-version/core/services.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.968525328Z namespaces/openshift-cluster-version/discovery.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.968590369Z namespaces/openshift-cluster-version/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.968668851Z namespaces/openshift-cluster-version/image.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.968722572Z namespaces/openshift-cluster-version/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.968776353Z namespaces/openshift-cluster-version/k8s.ovn.org/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.968811964Z namespaces/openshift-cluster-version/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.968895785Z namespaces/openshift-cluster-version/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.968969967Z namespaces/openshift-cluster-version/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.969039079Z namespaces/openshift-cluster-version/monitoring.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.969079689Z namespaces/openshift-cluster-version/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.969139001Z namespaces/openshift-cluster-version/networking.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.969171851Z namespaces/openshift-cluster-version/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.969233512Z namespaces/openshift-cluster-version/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.969240832Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.969285984Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.969549039Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.969766383Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.969904376Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.97009978Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.970334705Z namespaces/openshift-cluster-version/policy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.970363635Z namespaces/openshift-cluster-version/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.970432427Z namespaces/openshift-cluster-version/route.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.970459587Z namespaces/openshift-cluster-version/route.openshift.io/routes.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.97060693Z namespaces/openshift-config-managed/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.970649451Z namespaces/openshift-config-managed/openshift-config-managed.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.970780514Z namespaces/openshift-config-managed/apps.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.970834825Z namespaces/openshift-config-managed/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.970907726Z namespaces/openshift-config-managed/apps/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.970954057Z namespaces/openshift-config-managed/apps/daemonsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.971028689Z namespaces/openshift-config-managed/apps/deployments.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.97110919Z namespaces/openshift-config-managed/apps/replicasets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.971185992Z namespaces/openshift-config-managed/apps/statefulsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.971228553Z namespaces/openshift-config-managed/autoscaling/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.971272794Z namespaces/openshift-config-managed/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.971319925Z namespaces/openshift-config-managed/batch/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.971353806Z namespaces/openshift-config-managed/batch/cronjobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.971433257Z namespaces/openshift-config-managed/batch/jobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.971488658Z namespaces/openshift-config-managed/build.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.971528599Z namespaces/openshift-config-managed/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.971644811Z namespaces/openshift-config-managed/build.openshift.io/builds.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.971732263Z namespaces/openshift-config-managed/core/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.971788534Z namespaces/openshift-config-managed/core/configmaps.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.975317296Z namespaces/openshift-config-managed/core/endpoints.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.975392127Z namespaces/openshift-config-managed/core/events.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.975481479Z namespaces/openshift-config-managed/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.975568391Z namespaces/openshift-config-managed/core/pods.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.975753125Z namespaces/openshift-config-managed/core/replicationcontrollers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.975913038Z namespaces/openshift-config-managed/core/secrets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.97601386Z namespaces/openshift-config-managed/core/services.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.976177023Z namespaces/openshift-config-managed/core/configmaps/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.976207754Z namespaces/openshift-config-managed/core/configmaps/console-public.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.976307996Z namespaces/openshift-config-managed/core/configmaps/openshift-network-features.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.976345327Z namespaces/openshift-config-managed/discovery.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.976380517Z namespaces/openshift-config-managed/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.976431168Z namespaces/openshift-config-managed/image.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.976457839Z namespaces/openshift-config-managed/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.97652139Z namespaces/openshift-config-managed/k8s.ovn.org/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.976549861Z namespaces/openshift-config-managed/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.976684894Z namespaces/openshift-config-managed/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.976764295Z namespaces/openshift-config-managed/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.976814246Z namespaces/openshift-config-managed/monitoring.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.976853277Z namespaces/openshift-config-managed/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.976926268Z namespaces/openshift-config-managed/networking.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.976954659Z namespaces/openshift-config-managed/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.97702357Z namespaces/openshift-config-managed/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.977033911Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.977092652Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.977297136Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.97750195Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.977685944Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.977898218Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.978119313Z namespaces/openshift-config-managed/policy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.978141823Z namespaces/openshift-config-managed/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.978199684Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.978208545Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.978254755Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/rolebindings/openshift-network-public-role-binding.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.978328947Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.978368188Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/roles/openshift-network-public-role.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.978427259Z namespaces/openshift-config-managed/route.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.97847662Z namespaces/openshift-config-managed/route.openshift.io/routes.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.978525911Z namespaces/openshift-config-operator/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.978546211Z namespaces/openshift-config-operator/monitoring.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.978554041Z namespaces/openshift-config-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.978589232Z namespaces/openshift-config-operator/monitoring.coreos.com/servicemonitors/config-operator.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.978685824Z namespaces/openshift-config-operator/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.978696384Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.978747625Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.978902659Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.979091683Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.979226475Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.979425409Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.979829458Z namespaces/openshift-config/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.979879159Z namespaces/openshift-config/openshift-config.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.97993861Z namespaces/openshift-config/apps.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.979986711Z namespaces/openshift-config/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.980050752Z namespaces/openshift-config/apps/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.980086363Z namespaces/openshift-config/apps/daemonsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.980171914Z namespaces/openshift-config/apps/deployments.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.980248056Z namespaces/openshift-config/apps/replicasets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.980328637Z namespaces/openshift-config/apps/statefulsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.980382329Z namespaces/openshift-config/autoscaling/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.980417709Z namespaces/openshift-config/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.980480851Z namespaces/openshift-config/batch/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.980521561Z namespaces/openshift-config/batch/cronjobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.980632994Z namespaces/openshift-config/batch/jobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.980687345Z namespaces/openshift-config/build.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.980734326Z namespaces/openshift-config/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.980811157Z namespaces/openshift-config/build.openshift.io/builds.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.980856968Z namespaces/openshift-config/core/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.980888319Z namespaces/openshift-config/core/configmaps.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.981037352Z namespaces/openshift-config/core/endpoints.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.981111373Z namespaces/openshift-config/core/events.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.981188385Z namespaces/openshift-config/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.981272527Z namespaces/openshift-config/core/pods.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.981345048Z namespaces/openshift-config/core/replicationcontrollers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.981493431Z namespaces/openshift-config/core/secrets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.981672285Z namespaces/openshift-config/core/services.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.981729446Z namespaces/openshift-config/core/secrets/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.981778967Z namespaces/openshift-config/core/secrets/pull-secret.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.981827638Z namespaces/openshift-config/discovery.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.981880729Z namespaces/openshift-config/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.9819348Z namespaces/openshift-config/image.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.981962761Z namespaces/openshift-config/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.982036622Z namespaces/openshift-config/k8s.ovn.org/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.982074463Z namespaces/openshift-config/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.982161865Z namespaces/openshift-config/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.982234586Z namespaces/openshift-config/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.982282077Z namespaces/openshift-config/monitoring.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.982308258Z namespaces/openshift-config/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.982387019Z namespaces/openshift-config/networking.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.98241274Z namespaces/openshift-config/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.982478801Z namespaces/openshift-config/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.982488931Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.982553383Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.982741066Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.98292054Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.983074603Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.983269687Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.983733397Z namespaces/openshift-config/policy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.983777107Z namespaces/openshift-config/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.983827868Z namespaces/openshift-config/route.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.983865449Z namespaces/openshift-config/route.openshift.io/routes.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.983970911Z namespaces/openshift-console-operator/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.984010802Z namespaces/openshift-console-operator/openshift-console-operator.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.984124234Z namespaces/openshift-console-operator/apps.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.984152875Z namespaces/openshift-console-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.984238897Z namespaces/openshift-console-operator/apps/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.984266657Z namespaces/openshift-console-operator/apps/daemonsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.984356999Z namespaces/openshift-console-operator/apps/deployments.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.984475961Z namespaces/openshift-console-operator/apps/replicasets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.984586274Z namespaces/openshift-console-operator/apps/statefulsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.984653985Z namespaces/openshift-console-operator/autoscaling/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.984707356Z namespaces/openshift-console-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.984754087Z namespaces/openshift-console-operator/batch/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.984787978Z namespaces/openshift-console-operator/batch/cronjobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.9848775Z namespaces/openshift-console-operator/batch/jobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.984914501Z namespaces/openshift-console-operator/build.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.984952881Z namespaces/openshift-console-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.985033493Z namespaces/openshift-console-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.985127965Z namespaces/openshift-console-operator/coordination.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.985135245Z namespaces/openshift-console-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.985178006Z namespaces/openshift-console-operator/coordination.k8s.io/leases/console-operator-lock.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.985244457Z namespaces/openshift-console-operator/core/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.985288198Z namespaces/openshift-console-operator/core/configmaps.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.985911611Z namespaces/openshift-console-operator/core/endpoints.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.985996932Z namespaces/openshift-console-operator/core/events.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.986252288Z namespaces/openshift-console-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.986325779Z namespaces/openshift-console-operator/core/pods.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.986450582Z namespaces/openshift-console-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.986596365Z namespaces/openshift-console-operator/core/secrets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.986799119Z namespaces/openshift-console-operator/core/services.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.98685301Z namespaces/openshift-console-operator/discovery.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.986881881Z namespaces/openshift-console-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.986949242Z namespaces/openshift-console-operator/image.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.986976562Z namespaces/openshift-console-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.987043244Z namespaces/openshift-console-operator/k8s.ovn.org/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.987074034Z namespaces/openshift-console-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.987167286Z namespaces/openshift-console-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.987241258Z namespaces/openshift-console-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.987309479Z namespaces/openshift-console-operator/monitoring.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.98734937Z namespaces/openshift-console-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.987402351Z namespaces/openshift-console-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.987436462Z namespaces/openshift-console-operator/monitoring.coreos.com/prometheusrules/cluster-monitoring-prometheus-rules.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.987501603Z namespaces/openshift-console-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.987542354Z namespaces/openshift-console-operator/monitoring.coreos.com/servicemonitors/console-operator.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.987598725Z namespaces/openshift-console-operator/networking.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.987669146Z namespaces/openshift-console-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.987725407Z namespaces/openshift-console-operator/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.987732378Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.987778919Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.987941682Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.988117615Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.98831946Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.988519374Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.98882168Z namespaces/openshift-console-operator/pods/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.98883196Z namespaces/openshift-console-operator/pods/console-operator-584dcb6b54-g7s47/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.988866191Z namespaces/openshift-console-operator/pods/console-operator-584dcb6b54-g7s47/console-operator-584dcb6b54-g7s47.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.988968953Z namespaces/openshift-console-operator/pods/console-operator-584dcb6b54-g7s47/console-operator/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.988976533Z namespaces/openshift-console-operator/pods/console-operator-584dcb6b54-g7s47/console-operator/console-operator/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.988980693Z namespaces/openshift-console-operator/pods/console-operator-584dcb6b54-g7s47/console-operator/console-operator/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.989019934Z namespaces/openshift-console-operator/pods/console-operator-584dcb6b54-g7s47/console-operator/console-operator/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.989522904Z namespaces/openshift-console-operator/pods/console-operator-584dcb6b54-g7s47/console-operator/console-operator/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.989557225Z namespaces/openshift-console-operator/policy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.989642457Z namespaces/openshift-console-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.989714168Z namespaces/openshift-console-operator/route.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.989742959Z namespaces/openshift-console-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.98981135Z namespaces/openshift-console-user-settings/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.98981842Z namespaces/openshift-console-user-settings/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.98983047Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.989874101Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.990025554Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.990207688Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.990343631Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.990559535Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.99081003Z namespaces/openshift-console/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.990862311Z namespaces/openshift-console/openshift-console.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.991015944Z namespaces/openshift-console/apps.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.991056905Z namespaces/openshift-console/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.991114976Z namespaces/openshift-console/apps/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.991166757Z namespaces/openshift-console/apps/daemonsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.991251579Z namespaces/openshift-console/apps/deployments.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.991428413Z namespaces/openshift-console/apps/replicasets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.991882262Z namespaces/openshift-console/apps/statefulsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.991907052Z namespaces/openshift-console/autoscaling/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.991965204Z namespaces/openshift-console/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.992037515Z namespaces/openshift-console/batch/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.992053915Z namespaces/openshift-console/batch/cronjobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.992158137Z namespaces/openshift-console/batch/jobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.992207109Z namespaces/openshift-console/build.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.992253939Z namespaces/openshift-console/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.992330521Z namespaces/openshift-console/build.openshift.io/builds.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.992386242Z namespaces/openshift-console/core/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.992429313Z namespaces/openshift-console/core/configmaps.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.993066926Z namespaces/openshift-console/core/endpoints.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.993148118Z namespaces/openshift-console/core/events.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.993425173Z namespaces/openshift-console/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.993497345Z namespaces/openshift-console/core/pods.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.993723269Z namespaces/openshift-console/core/replicationcontrollers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.993850132Z namespaces/openshift-console/core/secrets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.994050986Z namespaces/openshift-console/core/services.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.994108427Z namespaces/openshift-console/discovery.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.994143848Z namespaces/openshift-console/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.994216309Z namespaces/openshift-console/image.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.99426471Z namespaces/openshift-console/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.994511955Z namespaces/openshift-console/k8s.ovn.org/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.994550506Z namespaces/openshift-console/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.994685769Z namespaces/openshift-console/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.99476286Z namespaces/openshift-console/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.994816902Z namespaces/openshift-console/monitoring.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.994855352Z namespaces/openshift-console/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.994916264Z namespaces/openshift-console/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.994943054Z namespaces/openshift-console/monitoring.coreos.com/servicemonitors/console.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.995010405Z namespaces/openshift-console/networking.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.995040466Z namespaces/openshift-console/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.995109697Z namespaces/openshift-console/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.995116657Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.995170469Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.995322872Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.995601517Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.99575088Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.995949894Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.996148798Z namespaces/openshift-console/pods/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.996157509Z namespaces/openshift-console/pods/console-598bbfdd7c-tb6nc/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.99620883Z namespaces/openshift-console/pods/console-598bbfdd7c-tb6nc/console-598bbfdd7c-tb6nc.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.996309022Z namespaces/openshift-console/pods/console-598bbfdd7c-tb6nc/console/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.996318142Z namespaces/openshift-console/pods/console-598bbfdd7c-tb6nc/console/console/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.996323052Z namespaces/openshift-console/pods/console-598bbfdd7c-tb6nc/console/console/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.996330612Z namespaces/openshift-console/pods/console-598bbfdd7c-tb6nc/console/console/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.996429904Z namespaces/openshift-console/pods/console-598bbfdd7c-tb6nc/console/console/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.996509896Z namespaces/openshift-console/pods/console-598bbfdd7c-tb6nc/console/console/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.996522866Z namespaces/openshift-console/pods/downloads-7c97dc4555-xktrq/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.996589558Z namespaces/openshift-console/pods/downloads-7c97dc4555-xktrq/downloads-7c97dc4555-xktrq.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.99670594Z namespaces/openshift-console/pods/downloads-7c97dc4555-xktrq/download-server/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.99671721Z namespaces/openshift-console/pods/downloads-7c97dc4555-xktrq/download-server/download-server/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.99672268Z namespaces/openshift-console/pods/downloads-7c97dc4555-xktrq/download-server/download-server/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.996756931Z namespaces/openshift-console/pods/downloads-7c97dc4555-xktrq/download-server/download-server/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.997172099Z namespaces/openshift-console/pods/downloads-7c97dc4555-xktrq/download-server/download-server/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.997237531Z namespaces/openshift-console/pods/downloads-7c97dc4555-xktrq/download-server/download-server/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.997265831Z namespaces/openshift-console/policy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.997294622Z namespaces/openshift-console/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.997361393Z namespaces/openshift-console/route.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.997402134Z namespaces/openshift-console/route.openshift.io/routes.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.997513256Z namespaces/openshift-controller-manager-operator/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.997560007Z namespaces/openshift-controller-manager-operator/openshift-controller-manager-operator.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.997654889Z namespaces/openshift-controller-manager-operator/apps.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.99770021Z namespaces/openshift-controller-manager-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.997740871Z namespaces/openshift-controller-manager-operator/apps/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.997786332Z namespaces/openshift-controller-manager-operator/apps/daemonsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.997870513Z namespaces/openshift-controller-manager-operator/apps/deployments.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.997958915Z namespaces/openshift-controller-manager-operator/apps/replicasets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.9981816Z namespaces/openshift-controller-manager-operator/apps/statefulsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.998230831Z namespaces/openshift-controller-manager-operator/autoscaling/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.998275852Z namespaces/openshift-controller-manager-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.998344783Z namespaces/openshift-controller-manager-operator/batch/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.998382874Z namespaces/openshift-controller-manager-operator/batch/cronjobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.998470236Z namespaces/openshift-controller-manager-operator/batch/jobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.998523937Z namespaces/openshift-controller-manager-operator/build.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.998587798Z namespaces/openshift-controller-manager-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.998743201Z namespaces/openshift-controller-manager-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.998813183Z namespaces/openshift-controller-manager-operator/core/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.998861844Z namespaces/openshift-controller-manager-operator/core/configmaps.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.998998706Z namespaces/openshift-controller-manager-operator/core/endpoints.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.999078538Z namespaces/openshift-controller-manager-operator/core/events.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.99916011Z namespaces/openshift-controller-manager-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.999235501Z namespaces/openshift-controller-manager-operator/core/pods.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.999317813Z namespaces/openshift-controller-manager-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.999441715Z namespaces/openshift-controller-manager-operator/core/secrets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.999729541Z namespaces/openshift-controller-manager-operator/core/services.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.999779832Z namespaces/openshift-controller-manager-operator/discovery.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.999828953Z namespaces/openshift-controller-manager-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.999896105Z namespaces/openshift-controller-manager-operator/image.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:08.999943126Z namespaces/openshift-controller-manager-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.000008067Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.000055418Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.000131139Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.000212931Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.000241361Z namespaces/openshift-controller-manager-operator/monitoring.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.000290613Z namespaces/openshift-controller-manager-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.000341174Z namespaces/openshift-controller-manager-operator/networking.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.000377514Z namespaces/openshift-controller-manager-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.000425525Z namespaces/openshift-controller-manager-operator/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.000438345Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.000606889Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.00064066Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.000833224Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.000969816Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.00116226Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.001354454Z namespaces/openshift-controller-manager-operator/policy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.001391455Z namespaces/openshift-controller-manager-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.001438956Z namespaces/openshift-controller-manager-operator/route.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.001469846Z namespaces/openshift-controller-manager-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.001562988Z namespaces/openshift-controller-manager/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.00163561Z namespaces/openshift-controller-manager/openshift-controller-manager.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.001712601Z namespaces/openshift-controller-manager/apps.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.001739572Z namespaces/openshift-controller-manager/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.001795313Z namespaces/openshift-controller-manager/apps/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.001839254Z namespaces/openshift-controller-manager/apps/daemonsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.001913035Z namespaces/openshift-controller-manager/apps/deployments.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.001994617Z namespaces/openshift-controller-manager/apps/replicasets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.002071849Z namespaces/openshift-controller-manager/apps/statefulsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.002159071Z namespaces/openshift-controller-manager/autoscaling/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.003081529Z namespaces/openshift-controller-manager/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.003098059Z namespaces/openshift-controller-manager/batch/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.003102579Z namespaces/openshift-controller-manager/batch/cronjobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.00310677Z namespaces/openshift-controller-manager/batch/jobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.00311054Z namespaces/openshift-controller-manager/build.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.00311444Z namespaces/openshift-controller-manager/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.0031183Z namespaces/openshift-controller-manager/build.openshift.io/builds.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.0031223Z namespaces/openshift-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.00312617Z namespaces/openshift-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.00313386Z namespaces/openshift-controller-manager/coordination.k8s.io/leases/openshift-master-controllers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.00313764Z namespaces/openshift-controller-manager/core/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.00314145Z namespaces/openshift-controller-manager/core/configmaps.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.00315048Z namespaces/openshift-controller-manager/core/endpoints.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.003154101Z namespaces/openshift-controller-manager/core/events.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.003157801Z namespaces/openshift-controller-manager/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.003168901Z namespaces/openshift-controller-manager/core/pods.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.003232482Z namespaces/openshift-controller-manager/core/replicationcontrollers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.003354785Z namespaces/openshift-controller-manager/core/secrets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.003511478Z namespaces/openshift-controller-manager/core/services.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.003823144Z namespaces/openshift-controller-manager/discovery.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.003868455Z namespaces/openshift-controller-manager/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.003910666Z namespaces/openshift-controller-manager/image.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.003943207Z namespaces/openshift-controller-manager/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.003995888Z namespaces/openshift-controller-manager/k8s.ovn.org/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.004026288Z namespaces/openshift-controller-manager/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.00412782Z namespaces/openshift-controller-manager/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.004214992Z namespaces/openshift-controller-manager/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.004273693Z namespaces/openshift-controller-manager/monitoring.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.004325854Z namespaces/openshift-controller-manager/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.004356785Z namespaces/openshift-controller-manager/networking.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.004403286Z namespaces/openshift-controller-manager/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.004436246Z namespaces/openshift-controller-manager/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.004446487Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.004489868Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.004687282Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.004870125Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.005006168Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.005197702Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.005396196Z namespaces/openshift-controller-manager/policy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.005429327Z namespaces/openshift-controller-manager/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.005480958Z namespaces/openshift-controller-manager/route.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.005514108Z namespaces/openshift-controller-manager/route.openshift.io/routes.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.005630911Z namespaces/openshift-dns-operator/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.005685042Z namespaces/openshift-dns-operator/openshift-dns-operator.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.005747343Z namespaces/openshift-dns-operator/apps.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.005781314Z namespaces/openshift-dns-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.005840205Z namespaces/openshift-dns-operator/apps/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.005867956Z namespaces/openshift-dns-operator/apps/daemonsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.005965588Z namespaces/openshift-dns-operator/apps/deployments.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.006040169Z namespaces/openshift-dns-operator/apps/replicasets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.006121821Z namespaces/openshift-dns-operator/apps/statefulsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.006172132Z namespaces/openshift-dns-operator/autoscaling/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.006214493Z namespaces/openshift-dns-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.006252554Z namespaces/openshift-dns-operator/batch/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.006294164Z namespaces/openshift-dns-operator/batch/cronjobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.006370336Z namespaces/openshift-dns-operator/batch/jobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.006424497Z namespaces/openshift-dns-operator/build.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.006465988Z namespaces/openshift-dns-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.006607571Z namespaces/openshift-dns-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.006678962Z namespaces/openshift-dns-operator/core/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.006722833Z namespaces/openshift-dns-operator/core/configmaps.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.006828555Z namespaces/openshift-dns-operator/core/endpoints.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.006903847Z namespaces/openshift-dns-operator/core/events.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.007006059Z namespaces/openshift-dns-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.00707904Z namespaces/openshift-dns-operator/core/pods.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.007160562Z namespaces/openshift-dns-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.007275134Z namespaces/openshift-dns-operator/core/secrets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.007453058Z namespaces/openshift-dns-operator/core/services.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.007523129Z namespaces/openshift-dns-operator/discovery.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.00756822Z namespaces/openshift-dns-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.007651772Z namespaces/openshift-dns-operator/image.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.007700803Z namespaces/openshift-dns-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.007738654Z namespaces/openshift-dns-operator/k8s.ovn.org/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.007792445Z namespaces/openshift-dns-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.007867106Z namespaces/openshift-dns-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.007942938Z namespaces/openshift-dns-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.007993009Z namespaces/openshift-dns-operator/monitoring.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.008008269Z namespaces/openshift-dns-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.00807051Z namespaces/openshift-dns-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.008122412Z namespaces/openshift-dns-operator/monitoring.coreos.com/prometheusrules/dns.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.008172653Z namespaces/openshift-dns-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.008215553Z namespaces/openshift-dns-operator/monitoring.coreos.com/servicemonitors/dns-operator.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.008263344Z namespaces/openshift-dns-operator/networking.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.008307875Z namespaces/openshift-dns-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.008346916Z namespaces/openshift-dns-operator/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.008354316Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.008385667Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.00854336Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.008858036Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.008997609Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.009191453Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.009390727Z namespaces/openshift-dns-operator/policy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.009428958Z namespaces/openshift-dns-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.009493979Z namespaces/openshift-dns-operator/route.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.00952175Z namespaces/openshift-dns-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.009657583Z namespaces/openshift-dns/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.009704313Z namespaces/openshift-dns/openshift-dns.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.009769215Z namespaces/openshift-dns/apps.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.009809166Z namespaces/openshift-dns/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.010638472Z namespaces/openshift-dns/apps/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.010676993Z namespaces/openshift-dns/apps/daemonsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.010830516Z namespaces/openshift-dns/apps/deployments.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.010906718Z namespaces/openshift-dns/apps/replicasets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.01098958Z namespaces/openshift-dns/apps/statefulsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.01101885Z namespaces/openshift-dns/autoscaling/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.011056711Z namespaces/openshift-dns/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.011128853Z namespaces/openshift-dns/batch/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.011166593Z namespaces/openshift-dns/batch/cronjobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.011253345Z namespaces/openshift-dns/batch/jobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.011312146Z namespaces/openshift-dns/build.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.011355577Z namespaces/openshift-dns/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.011436429Z namespaces/openshift-dns/build.openshift.io/builds.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.01150573Z namespaces/openshift-dns/core/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.011550621Z namespaces/openshift-dns/core/configmaps.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.011694484Z namespaces/openshift-dns/core/endpoints.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.011778615Z namespaces/openshift-dns/core/events.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.01200326Z namespaces/openshift-dns/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.012077372Z namespaces/openshift-dns/core/pods.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.012348777Z namespaces/openshift-dns/core/replicationcontrollers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.012466869Z namespaces/openshift-dns/core/secrets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.012664873Z namespaces/openshift-dns/core/services.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.012733505Z namespaces/openshift-dns/discovery.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.012764126Z namespaces/openshift-dns/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.012842317Z namespaces/openshift-dns/image.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.012886528Z namespaces/openshift-dns/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.012948059Z namespaces/openshift-dns/k8s.ovn.org/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.01299407Z namespaces/openshift-dns/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.013081802Z namespaces/openshift-dns/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.013269636Z namespaces/openshift-dns/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.013335827Z namespaces/openshift-dns/monitoring.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.013364038Z namespaces/openshift-dns/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.013426279Z namespaces/openshift-dns/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.0134627Z namespaces/openshift-dns/monitoring.coreos.com/servicemonitors/dns-default.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.013516981Z namespaces/openshift-dns/networking.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.013545142Z namespaces/openshift-dns/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.013628333Z namespaces/openshift-dns/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.013639813Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.013665874Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.013823997Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.014006631Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.014137063Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.014327447Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.014625403Z namespaces/openshift-dns/pods/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.014638864Z namespaces/openshift-dns/pods/dns-default-nsq8v/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.014728015Z namespaces/openshift-dns/pods/dns-default-nsq8v/dns-default-nsq8v.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.014825067Z namespaces/openshift-dns/pods/dns-default-nsq8v/dns/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.014832508Z namespaces/openshift-dns/pods/dns-default-nsq8v/dns/dns/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.014845618Z namespaces/openshift-dns/pods/dns-default-nsq8v/dns/dns/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.014888399Z namespaces/openshift-dns/pods/dns-default-nsq8v/dns/dns/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.014989871Z namespaces/openshift-dns/pods/dns-default-nsq8v/dns/dns/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.015052332Z namespaces/openshift-dns/pods/dns-default-nsq8v/dns/dns/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.015083743Z namespaces/openshift-dns/pods/dns-default-nsq8v/kube-rbac-proxy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.015112133Z namespaces/openshift-dns/pods/dns-default-nsq8v/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.015120463Z namespaces/openshift-dns/pods/dns-default-nsq8v/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.015156154Z namespaces/openshift-dns/pods/dns-default-nsq8v/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.015311337Z namespaces/openshift-dns/pods/dns-default-nsq8v/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.015386879Z namespaces/openshift-dns/pods/dns-default-nsq8v/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.01542496Z namespaces/openshift-dns/pods/dns-default-thhvn/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.015531352Z namespaces/openshift-dns/pods/dns-default-thhvn/dns-default-thhvn.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.015647704Z namespaces/openshift-dns/pods/dns-default-thhvn/dns/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.015659284Z namespaces/openshift-dns/pods/dns-default-thhvn/dns/dns/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.015666124Z namespaces/openshift-dns/pods/dns-default-thhvn/dns/dns/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.015710755Z namespaces/openshift-dns/pods/dns-default-thhvn/dns/dns/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.015804347Z namespaces/openshift-dns/pods/dns-default-thhvn/dns/dns/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.015870578Z namespaces/openshift-dns/pods/dns-default-thhvn/dns/dns/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.01592492Z namespaces/openshift-dns/pods/dns-default-thhvn/kube-rbac-proxy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.01593212Z namespaces/openshift-dns/pods/dns-default-thhvn/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.01593652Z namespaces/openshift-dns/pods/dns-default-thhvn/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.015976631Z namespaces/openshift-dns/pods/dns-default-thhvn/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.016074033Z namespaces/openshift-dns/pods/dns-default-thhvn/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.016144034Z namespaces/openshift-dns/pods/dns-default-thhvn/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.016202805Z namespaces/openshift-dns/pods/dns-default-x9gk6/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.016230116Z namespaces/openshift-dns/pods/dns-default-x9gk6/dns-default-x9gk6.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.016323008Z namespaces/openshift-dns/pods/dns-default-x9gk6/dns/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.016330748Z namespaces/openshift-dns/pods/dns-default-x9gk6/dns/dns/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.016340248Z namespaces/openshift-dns/pods/dns-default-x9gk6/dns/dns/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.016387879Z namespaces/openshift-dns/pods/dns-default-x9gk6/dns/dns/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.016484841Z namespaces/openshift-dns/pods/dns-default-x9gk6/dns/dns/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.016549732Z namespaces/openshift-dns/pods/dns-default-x9gk6/dns/dns/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.016599744Z namespaces/openshift-dns/pods/dns-default-x9gk6/kube-rbac-proxy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.016614554Z namespaces/openshift-dns/pods/dns-default-x9gk6/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.016624124Z namespaces/openshift-dns/pods/dns-default-x9gk6/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.016685595Z namespaces/openshift-dns/pods/dns-default-x9gk6/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.016773657Z namespaces/openshift-dns/pods/dns-default-x9gk6/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.016845608Z namespaces/openshift-dns/pods/dns-default-x9gk6/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.016873059Z namespaces/openshift-dns/pods/node-resolver-2rf9h/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.01690479Z namespaces/openshift-dns/pods/node-resolver-2rf9h/node-resolver-2rf9h.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.017008062Z namespaces/openshift-dns/pods/node-resolver-2rf9h/dns-node-resolver/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.017015782Z namespaces/openshift-dns/pods/node-resolver-2rf9h/dns-node-resolver/dns-node-resolver/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.017020042Z namespaces/openshift-dns/pods/node-resolver-2rf9h/dns-node-resolver/dns-node-resolver/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.017064333Z namespaces/openshift-dns/pods/node-resolver-2rf9h/dns-node-resolver/dns-node-resolver/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.017180315Z namespaces/openshift-dns/pods/node-resolver-2rf9h/dns-node-resolver/dns-node-resolver/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.017252287Z namespaces/openshift-dns/pods/node-resolver-2rf9h/dns-node-resolver/dns-node-resolver/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.017302108Z namespaces/openshift-dns/pods/node-resolver-5z8xh/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.017333288Z namespaces/openshift-dns/pods/node-resolver-5z8xh/node-resolver-5z8xh.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.01743807Z namespaces/openshift-dns/pods/node-resolver-5z8xh/dns-node-resolver/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.01744673Z namespaces/openshift-dns/pods/node-resolver-5z8xh/dns-node-resolver/dns-node-resolver/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.017450881Z namespaces/openshift-dns/pods/node-resolver-5z8xh/dns-node-resolver/dns-node-resolver/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.017486321Z namespaces/openshift-dns/pods/node-resolver-5z8xh/dns-node-resolver/dns-node-resolver/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.017594633Z namespaces/openshift-dns/pods/node-resolver-5z8xh/dns-node-resolver/dns-node-resolver/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.017682985Z namespaces/openshift-dns/pods/node-resolver-5z8xh/dns-node-resolver/dns-node-resolver/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.017713166Z namespaces/openshift-dns/pods/node-resolver-hpntr/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.017748257Z namespaces/openshift-dns/pods/node-resolver-hpntr/node-resolver-hpntr.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.017835969Z namespaces/openshift-dns/pods/node-resolver-hpntr/dns-node-resolver/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.017843398Z namespaces/openshift-dns/pods/node-resolver-hpntr/dns-node-resolver/dns-node-resolver/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.017847569Z namespaces/openshift-dns/pods/node-resolver-hpntr/dns-node-resolver/dns-node-resolver/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.017871039Z namespaces/openshift-dns/pods/node-resolver-hpntr/dns-node-resolver/dns-node-resolver/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.017987612Z namespaces/openshift-dns/pods/node-resolver-hpntr/dns-node-resolver/dns-node-resolver/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.018058793Z namespaces/openshift-dns/pods/node-resolver-hpntr/dns-node-resolver/dns-node-resolver/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.018100354Z namespaces/openshift-dns/policy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.018135535Z namespaces/openshift-dns/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.018193936Z namespaces/openshift-dns/route.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.018261827Z namespaces/openshift-dns/route.openshift.io/routes.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.018358369Z namespaces/openshift-etcd/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.0184071Z namespaces/openshift-etcd/openshift-etcd.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.018464511Z namespaces/openshift-etcd/apps.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.018510322Z namespaces/openshift-etcd/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.018590604Z namespaces/openshift-etcd/apps/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.018673655Z namespaces/openshift-etcd/apps/daemonsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.018761857Z namespaces/openshift-etcd/apps/deployments.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.018851089Z namespaces/openshift-etcd/apps/replicasets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.018930671Z namespaces/openshift-etcd/apps/statefulsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.018983082Z namespaces/openshift-etcd/autoscaling/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.019000572Z namespaces/openshift-etcd/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.019062553Z namespaces/openshift-etcd/batch/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.019110024Z namespaces/openshift-etcd/batch/cronjobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.019184336Z namespaces/openshift-etcd/batch/jobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.019243647Z namespaces/openshift-etcd/build.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.019271798Z namespaces/openshift-etcd/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.01936266Z namespaces/openshift-etcd/build.openshift.io/builds.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.019434101Z namespaces/openshift-etcd/core/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.019463031Z namespaces/openshift-etcd/core/configmaps.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.019592484Z namespaces/openshift-etcd/core/endpoints.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.019827179Z namespaces/openshift-etcd/core/events.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.01990096Z namespaces/openshift-etcd/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.019980902Z namespaces/openshift-etcd/core/pods.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.020062724Z namespaces/openshift-etcd/core/replicationcontrollers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.020180046Z namespaces/openshift-etcd/core/secrets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.020330819Z namespaces/openshift-etcd/core/services.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.02039213Z namespaces/openshift-etcd/discovery.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.020425391Z namespaces/openshift-etcd/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.020480322Z namespaces/openshift-etcd/image.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.020511973Z namespaces/openshift-etcd/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.020565674Z namespaces/openshift-etcd/k8s.ovn.org/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.020626735Z namespaces/openshift-etcd/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.020714957Z namespaces/openshift-etcd/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.020788948Z namespaces/openshift-etcd/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.02085258Z namespaces/openshift-etcd/monitoring.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.02088692Z namespaces/openshift-etcd/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.020939492Z namespaces/openshift-etcd/networking.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.020977522Z namespaces/openshift-etcd/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.021024323Z namespaces/openshift-etcd/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.021034283Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.021088015Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.021252488Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.021452292Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.021592015Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.021934702Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.022131076Z namespaces/openshift-etcd/policy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.022159816Z namespaces/openshift-etcd/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.022228058Z namespaces/openshift-etcd/route.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.022261798Z namespaces/openshift-etcd/route.openshift.io/routes.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.022382441Z namespaces/openshift-host-network/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.022415491Z namespaces/openshift-host-network/openshift-host-network.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.022481903Z namespaces/openshift-host-network/apps.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.022514464Z namespaces/openshift-host-network/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.022626456Z namespaces/openshift-host-network/apps/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.022663116Z namespaces/openshift-host-network/apps/daemonsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.022754748Z namespaces/openshift-host-network/apps/deployments.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.02283735Z namespaces/openshift-host-network/apps/replicasets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.022910172Z namespaces/openshift-host-network/apps/statefulsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.022969903Z namespaces/openshift-host-network/autoscaling/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.022993663Z namespaces/openshift-host-network/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.023063884Z namespaces/openshift-host-network/batch/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.023103825Z namespaces/openshift-host-network/batch/cronjobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.023190137Z namespaces/openshift-host-network/batch/jobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.023241278Z namespaces/openshift-host-network/build.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.023284899Z namespaces/openshift-host-network/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.023364811Z namespaces/openshift-host-network/build.openshift.io/builds.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.023441812Z namespaces/openshift-host-network/core/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.023475853Z namespaces/openshift-host-network/core/configmaps.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.023586985Z namespaces/openshift-host-network/core/endpoints.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.023680207Z namespaces/openshift-host-network/core/events.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.023757458Z namespaces/openshift-host-network/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.02383482Z namespaces/openshift-host-network/core/pods.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.023915592Z namespaces/openshift-host-network/core/replicationcontrollers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.024055535Z namespaces/openshift-host-network/core/secrets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.024173207Z namespaces/openshift-host-network/core/services.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.024206838Z namespaces/openshift-host-network/core/resourcequotas/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.024259059Z namespaces/openshift-host-network/core/resourcequotas/host-network-namespace-quotas.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.02432623Z namespaces/openshift-host-network/discovery.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.024374231Z namespaces/openshift-host-network/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.024428532Z namespaces/openshift-host-network/image.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.024458003Z namespaces/openshift-host-network/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.024532734Z namespaces/openshift-host-network/k8s.ovn.org/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.024596136Z namespaces/openshift-host-network/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.024675477Z namespaces/openshift-host-network/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.024756799Z namespaces/openshift-host-network/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.02480692Z namespaces/openshift-host-network/monitoring.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.02482592Z namespaces/openshift-host-network/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.024884212Z namespaces/openshift-host-network/networking.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.024915352Z namespaces/openshift-host-network/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.024960803Z namespaces/openshift-host-network/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.024974753Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.025030404Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.025184447Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.025363401Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.025498484Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.025715528Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.025915422Z namespaces/openshift-host-network/policy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.025944783Z namespaces/openshift-host-network/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.025985574Z namespaces/openshift-host-network/route.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.026024925Z namespaces/openshift-host-network/route.openshift.io/routes.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.026112136Z namespaces/openshift-image-registry/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.026148497Z namespaces/openshift-image-registry/openshift-image-registry.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.026218859Z namespaces/openshift-image-registry/apps.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.026244139Z namespaces/openshift-image-registry/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.02630152Z namespaces/openshift-image-registry/apps/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.026334231Z namespaces/openshift-image-registry/apps/daemonsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.026432743Z namespaces/openshift-image-registry/apps/deployments.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.026599216Z namespaces/openshift-image-registry/apps/replicasets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.02680315Z namespaces/openshift-image-registry/apps/statefulsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.026821331Z namespaces/openshift-image-registry/autoscaling/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.026882832Z namespaces/openshift-image-registry/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.026919053Z namespaces/openshift-image-registry/batch/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.026969114Z namespaces/openshift-image-registry/batch/cronjobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.027070666Z namespaces/openshift-image-registry/batch/jobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.027129157Z namespaces/openshift-image-registry/build.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.027158638Z namespaces/openshift-image-registry/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.02725356Z namespaces/openshift-image-registry/build.openshift.io/builds.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.02728552Z namespaces/openshift-image-registry/coordination.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.027295361Z namespaces/openshift-image-registry/coordination.k8s.io/leases/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.027337941Z namespaces/openshift-image-registry/coordination.k8s.io/leases/openshift-master-controllers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.027420363Z namespaces/openshift-image-registry/core/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.027453814Z namespaces/openshift-image-registry/core/configmaps.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.028478304Z namespaces/openshift-image-registry/core/endpoints.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.028598437Z namespaces/openshift-image-registry/core/events.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.028860602Z namespaces/openshift-image-registry/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.028937474Z namespaces/openshift-image-registry/core/pods.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.029147498Z namespaces/openshift-image-registry/core/replicationcontrollers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.02926854Z namespaces/openshift-image-registry/core/secrets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.029449524Z namespaces/openshift-image-registry/core/services.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.029508115Z namespaces/openshift-image-registry/discovery.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.029535906Z namespaces/openshift-image-registry/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.029621717Z namespaces/openshift-image-registry/image.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.029665218Z namespaces/openshift-image-registry/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.029699319Z namespaces/openshift-image-registry/k8s.ovn.org/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.02974998Z namespaces/openshift-image-registry/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.029836312Z namespaces/openshift-image-registry/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.029912494Z namespaces/openshift-image-registry/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.029983515Z namespaces/openshift-image-registry/monitoring.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.030021206Z namespaces/openshift-image-registry/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.030092557Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.030133918Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/image-registry-operator-alerts.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.03023684Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/image-registry-rules.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.030322612Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/imagestreams-rules.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.030388843Z namespaces/openshift-image-registry/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.030422964Z namespaces/openshift-image-registry/monitoring.coreos.com/servicemonitors/image-registry.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.030482585Z namespaces/openshift-image-registry/networking.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.030529256Z namespaces/openshift-image-registry/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.030591107Z namespaces/openshift-image-registry/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.030641388Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.030673549Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.030840512Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.031017906Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.031163339Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.031374393Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.031560657Z namespaces/openshift-image-registry/pods/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.031567677Z namespaces/openshift-image-registry/pods/image-registry-64cd7c8686-n6ktg/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.031624078Z namespaces/openshift-image-registry/pods/image-registry-64cd7c8686-n6ktg/image-registry-64cd7c8686-n6ktg.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.0317354Z namespaces/openshift-image-registry/pods/image-registry-64cd7c8686-n6ktg/registry/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.031746321Z namespaces/openshift-image-registry/pods/image-registry-64cd7c8686-n6ktg/registry/registry/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.031756431Z namespaces/openshift-image-registry/pods/image-registry-64cd7c8686-n6ktg/registry/registry/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.031788501Z namespaces/openshift-image-registry/pods/image-registry-64cd7c8686-n6ktg/registry/registry/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.034133189Z namespaces/openshift-image-registry/pods/image-registry-64cd7c8686-n6ktg/registry/registry/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.03419532Z namespaces/openshift-image-registry/pods/image-registry-64cd7c8686-n6ktg/registry/registry/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.034231251Z namespaces/openshift-image-registry/pods/node-ca-d48vp/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.034267342Z namespaces/openshift-image-registry/pods/node-ca-d48vp/node-ca-d48vp.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.034355534Z namespaces/openshift-image-registry/pods/node-ca-d48vp/node-ca/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.034365254Z namespaces/openshift-image-registry/pods/node-ca-d48vp/node-ca/node-ca/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.034370384Z namespaces/openshift-image-registry/pods/node-ca-d48vp/node-ca/node-ca/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.034378324Z namespaces/openshift-image-registry/pods/node-ca-d48vp/node-ca/node-ca/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.034599698Z namespaces/openshift-image-registry/pods/node-ca-d48vp/node-ca/node-ca/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.03466592Z namespaces/openshift-image-registry/pods/node-ca-d48vp/node-ca/node-ca/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.034704831Z namespaces/openshift-image-registry/pods/node-ca-h8krb/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.034735981Z namespaces/openshift-image-registry/pods/node-ca-h8krb/node-ca-h8krb.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.034813593Z namespaces/openshift-image-registry/pods/node-ca-h8krb/node-ca/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.034824343Z namespaces/openshift-image-registry/pods/node-ca-h8krb/node-ca/node-ca/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.034835403Z namespaces/openshift-image-registry/pods/node-ca-h8krb/node-ca/node-ca/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.034885214Z namespaces/openshift-image-registry/pods/node-ca-h8krb/node-ca/node-ca/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.035058668Z namespaces/openshift-image-registry/pods/node-ca-h8krb/node-ca/node-ca/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.035124669Z namespaces/openshift-image-registry/pods/node-ca-h8krb/node-ca/node-ca/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.03515135Z namespaces/openshift-image-registry/pods/node-ca-mkf4n/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.035198581Z namespaces/openshift-image-registry/pods/node-ca-mkf4n/node-ca-mkf4n.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.035268692Z namespaces/openshift-image-registry/pods/node-ca-mkf4n/node-ca/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.035276242Z namespaces/openshift-image-registry/pods/node-ca-mkf4n/node-ca/node-ca/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.035287903Z namespaces/openshift-image-registry/pods/node-ca-mkf4n/node-ca/node-ca/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.035330813Z namespaces/openshift-image-registry/pods/node-ca-mkf4n/node-ca/node-ca/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.035513257Z namespaces/openshift-image-registry/pods/node-ca-mkf4n/node-ca/node-ca/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.035589789Z namespaces/openshift-image-registry/pods/node-ca-mkf4n/node-ca/node-ca/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.035689381Z namespaces/openshift-image-registry/policy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.035722861Z namespaces/openshift-image-registry/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.035790713Z namespaces/openshift-image-registry/route.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.035830274Z namespaces/openshift-image-registry/route.openshift.io/routes.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.035881145Z namespaces/openshift-infra/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.035890995Z namespaces/openshift-infra/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.035896915Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.035907605Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.036070868Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.036249812Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.036385385Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.036596039Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.036857324Z namespaces/openshift-ingress-canary/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.036907195Z namespaces/openshift-ingress-canary/openshift-ingress-canary.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.036977167Z namespaces/openshift-ingress-canary/apps.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.037024398Z namespaces/openshift-ingress-canary/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.037080499Z namespaces/openshift-ingress-canary/apps/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.03711945Z namespaces/openshift-ingress-canary/apps/daemonsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.037245902Z namespaces/openshift-ingress-canary/apps/deployments.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.037325364Z namespaces/openshift-ingress-canary/apps/replicasets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.037404066Z namespaces/openshift-ingress-canary/apps/statefulsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.037448786Z namespaces/openshift-ingress-canary/autoscaling/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.037489647Z namespaces/openshift-ingress-canary/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.037558858Z namespaces/openshift-ingress-canary/batch/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.0376176Z namespaces/openshift-ingress-canary/batch/cronjobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.037704261Z namespaces/openshift-ingress-canary/batch/jobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.037749762Z namespaces/openshift-ingress-canary/build.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.037781653Z namespaces/openshift-ingress-canary/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.037865495Z namespaces/openshift-ingress-canary/build.openshift.io/builds.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.037947206Z namespaces/openshift-ingress-canary/core/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.037978827Z namespaces/openshift-ingress-canary/core/configmaps.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.038093769Z namespaces/openshift-ingress-canary/core/endpoints.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.038178371Z namespaces/openshift-ingress-canary/core/events.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.038326794Z namespaces/openshift-ingress-canary/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.038399326Z namespaces/openshift-ingress-canary/core/pods.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.03859415Z namespaces/openshift-ingress-canary/core/replicationcontrollers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.038715372Z namespaces/openshift-ingress-canary/core/secrets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.038875885Z namespaces/openshift-ingress-canary/core/services.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.038928837Z namespaces/openshift-ingress-canary/discovery.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.038976827Z namespaces/openshift-ingress-canary/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.039026448Z namespaces/openshift-ingress-canary/image.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.039059639Z namespaces/openshift-ingress-canary/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.039139701Z namespaces/openshift-ingress-canary/k8s.ovn.org/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.039177711Z namespaces/openshift-ingress-canary/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.039264873Z namespaces/openshift-ingress-canary/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.039341285Z namespaces/openshift-ingress-canary/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.039389796Z namespaces/openshift-ingress-canary/monitoring.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.039420536Z namespaces/openshift-ingress-canary/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.039476737Z namespaces/openshift-ingress-canary/networking.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.039515158Z namespaces/openshift-ingress-canary/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.03958649Z namespaces/openshift-ingress-canary/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.03960303Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.039662811Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.039829565Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.040021619Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.040156411Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.040350775Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.040630981Z namespaces/openshift-ingress-canary/pods/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.040645521Z namespaces/openshift-ingress-canary/pods/ingress-canary-287lq/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.040671952Z namespaces/openshift-ingress-canary/pods/ingress-canary-287lq/ingress-canary-287lq.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.040762003Z namespaces/openshift-ingress-canary/pods/ingress-canary-287lq/serve-healthcheck-canary/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.040769604Z namespaces/openshift-ingress-canary/pods/ingress-canary-287lq/serve-healthcheck-canary/serve-healthcheck-canary/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.040779414Z namespaces/openshift-ingress-canary/pods/ingress-canary-287lq/serve-healthcheck-canary/serve-healthcheck-canary/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.040824505Z namespaces/openshift-ingress-canary/pods/ingress-canary-287lq/serve-healthcheck-canary/serve-healthcheck-canary/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.040926087Z namespaces/openshift-ingress-canary/pods/ingress-canary-287lq/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.040991498Z namespaces/openshift-ingress-canary/pods/ingress-canary-287lq/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.041034679Z namespaces/openshift-ingress-canary/pods/ingress-canary-2qvgv/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.04108287Z namespaces/openshift-ingress-canary/pods/ingress-canary-2qvgv/ingress-canary-2qvgv.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.041171062Z namespaces/openshift-ingress-canary/pods/ingress-canary-2qvgv/serve-healthcheck-canary/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.041180832Z namespaces/openshift-ingress-canary/pods/ingress-canary-2qvgv/serve-healthcheck-canary/serve-healthcheck-canary/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.041187242Z namespaces/openshift-ingress-canary/pods/ingress-canary-2qvgv/serve-healthcheck-canary/serve-healthcheck-canary/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.041225463Z namespaces/openshift-ingress-canary/pods/ingress-canary-2qvgv/serve-healthcheck-canary/serve-healthcheck-canary/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.041323645Z namespaces/openshift-ingress-canary/pods/ingress-canary-2qvgv/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.041396756Z namespaces/openshift-ingress-canary/pods/ingress-canary-2qvgv/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.041656182Z namespaces/openshift-ingress-canary/pods/ingress-canary-ljlnk/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.041701543Z namespaces/openshift-ingress-canary/pods/ingress-canary-ljlnk/ingress-canary-ljlnk.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.041777744Z namespaces/openshift-ingress-canary/pods/ingress-canary-ljlnk/serve-healthcheck-canary/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.041793435Z namespaces/openshift-ingress-canary/pods/ingress-canary-ljlnk/serve-healthcheck-canary/serve-healthcheck-canary/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.041806845Z namespaces/openshift-ingress-canary/pods/ingress-canary-ljlnk/serve-healthcheck-canary/serve-healthcheck-canary/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.041842365Z namespaces/openshift-ingress-canary/pods/ingress-canary-ljlnk/serve-healthcheck-canary/serve-healthcheck-canary/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.041943978Z namespaces/openshift-ingress-canary/pods/ingress-canary-ljlnk/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.041995609Z namespaces/openshift-ingress-canary/pods/ingress-canary-ljlnk/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.042023179Z namespaces/openshift-ingress-canary/policy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.04206429Z namespaces/openshift-ingress-canary/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.042127151Z namespaces/openshift-ingress-canary/route.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.042164402Z namespaces/openshift-ingress-canary/route.openshift.io/routes.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.042281514Z namespaces/openshift-ingress-operator/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.042327145Z namespaces/openshift-ingress-operator/openshift-ingress-operator.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.042399597Z namespaces/openshift-ingress-operator/apps.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.042438598Z namespaces/openshift-ingress-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.042504639Z namespaces/openshift-ingress-operator/apps/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.04254093Z namespaces/openshift-ingress-operator/apps/daemonsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.042820055Z namespaces/openshift-ingress-operator/apps/deployments.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.042894747Z namespaces/openshift-ingress-operator/apps/replicasets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.042974219Z namespaces/openshift-ingress-operator/apps/statefulsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.04303566Z namespaces/openshift-ingress-operator/autoscaling/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.04306222Z namespaces/openshift-ingress-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.043140002Z namespaces/openshift-ingress-operator/batch/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.043178193Z namespaces/openshift-ingress-operator/batch/cronjobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.043262314Z namespaces/openshift-ingress-operator/batch/jobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.043295185Z namespaces/openshift-ingress-operator/build.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.043342096Z namespaces/openshift-ingress-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.043425057Z namespaces/openshift-ingress-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.043500139Z namespaces/openshift-ingress-operator/core/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.043522519Z namespaces/openshift-ingress-operator/core/configmaps.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.044433508Z namespaces/openshift-ingress-operator/core/endpoints.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.04455046Z namespaces/openshift-ingress-operator/core/events.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.044704084Z namespaces/openshift-ingress-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.044780645Z namespaces/openshift-ingress-operator/core/pods.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.044862547Z namespaces/openshift-ingress-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.044985649Z namespaces/openshift-ingress-operator/core/secrets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.045167873Z namespaces/openshift-ingress-operator/core/services.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.045223744Z namespaces/openshift-ingress-operator/discovery.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.045263885Z namespaces/openshift-ingress-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.045323196Z namespaces/openshift-ingress-operator/image.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.045373727Z namespaces/openshift-ingress-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.045431448Z namespaces/openshift-ingress-operator/ingress.operator.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.045440049Z namespaces/openshift-ingress-operator/ingress.operator.openshift.io/dnsrecords/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.045479939Z namespaces/openshift-ingress-operator/ingress.operator.openshift.io/dnsrecords/default-wildcard.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.045549961Z namespaces/openshift-ingress-operator/k8s.ovn.org/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.045591652Z namespaces/openshift-ingress-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.045711214Z namespaces/openshift-ingress-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.045784345Z namespaces/openshift-ingress-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.045852027Z namespaces/openshift-ingress-operator/monitoring.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.045892838Z namespaces/openshift-ingress-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.045950419Z namespaces/openshift-ingress-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.04599483Z namespaces/openshift-ingress-operator/monitoring.coreos.com/prometheusrules/ingress-operator.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.046077822Z namespaces/openshift-ingress-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.046115542Z namespaces/openshift-ingress-operator/monitoring.coreos.com/servicemonitors/ingress-operator.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.046173063Z namespaces/openshift-ingress-operator/networking.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.046217494Z namespaces/openshift-ingress-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.046281615Z namespaces/openshift-ingress-operator/operator.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.046289536Z namespaces/openshift-ingress-operator/operator.openshift.io/ingresscontrollers/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.046306536Z namespaces/openshift-ingress-operator/operator.openshift.io/ingresscontrollers/default.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.046399488Z namespaces/openshift-ingress-operator/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.046414118Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.046459059Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.046651263Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.046831617Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.0469762Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.047176914Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.047375128Z namespaces/openshift-ingress-operator/policy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.047414929Z namespaces/openshift-ingress-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.04747326Z namespaces/openshift-ingress-operator/route.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.047516501Z namespaces/openshift-ingress-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.047775046Z namespaces/openshift-ingress/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.047816627Z namespaces/openshift-ingress/openshift-ingress.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.04799102Z namespaces/openshift-ingress/apps.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.048022361Z namespaces/openshift-ingress/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.048096463Z namespaces/openshift-ingress/apps/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.048126733Z namespaces/openshift-ingress/apps/daemonsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.048219955Z namespaces/openshift-ingress/apps/deployments.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.048446459Z namespaces/openshift-ingress/apps/replicasets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.048662694Z namespaces/openshift-ingress/apps/statefulsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.048713515Z namespaces/openshift-ingress/autoscaling/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.048771856Z namespaces/openshift-ingress/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.048815627Z namespaces/openshift-ingress/batch/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.048847238Z namespaces/openshift-ingress/batch/cronjobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.04893386Z namespaces/openshift-ingress/batch/jobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.048998951Z namespaces/openshift-ingress/build.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.049039891Z namespaces/openshift-ingress/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.049119783Z namespaces/openshift-ingress/build.openshift.io/builds.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.049175694Z namespaces/openshift-ingress/coordination.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.049185894Z namespaces/openshift-ingress/coordination.k8s.io/leases/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.049226965Z namespaces/openshift-ingress/coordination.k8s.io/leases/istio-gateway-ca-openshift-gateway.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.049319597Z namespaces/openshift-ingress/coordination.k8s.io/leases/istio-gateway-deployment-openshift-gateway.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.049401559Z namespaces/openshift-ingress/coordination.k8s.io/leases/istio-gateway-status-leader-openshift-gateway.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.049509601Z namespaces/openshift-ingress/core/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.049545672Z namespaces/openshift-ingress/core/configmaps.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.04997063Z namespaces/openshift-ingress/core/endpoints.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.050065192Z namespaces/openshift-ingress/core/events.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.050299637Z namespaces/openshift-ingress/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.050375059Z namespaces/openshift-ingress/core/pods.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.050664235Z namespaces/openshift-ingress/core/replicationcontrollers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.050779117Z namespaces/openshift-ingress/core/secrets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.050975331Z namespaces/openshift-ingress/core/services.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.051071903Z namespaces/openshift-ingress/discovery.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.051097603Z namespaces/openshift-ingress/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.051204315Z namespaces/openshift-ingress/gateway.networking.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.051216436Z namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.051230836Z namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/openshift-ai-inference.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.051317478Z namespaces/openshift-ingress/image.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.051352249Z namespaces/openshift-ingress/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.05142636Z namespaces/openshift-ingress/k8s.ovn.org/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.051460401Z namespaces/openshift-ingress/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.051546982Z namespaces/openshift-ingress/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.051680285Z namespaces/openshift-ingress/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.051736666Z namespaces/openshift-ingress/monitoring.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.051780627Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.051822998Z namespaces/openshift-ingress/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.051883059Z namespaces/openshift-ingress/monitoring.coreos.com/podmonitors/openshift-ai-inference-metrics.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.05194039Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.051991862Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors/router-default.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.052026492Z namespaces/openshift-ingress/networking.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.052060153Z namespaces/openshift-ingress/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.052120314Z namespaces/openshift-ingress/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.052134184Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.052181815Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.052331608Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.052536573Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.052709366Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.052933661Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.053157195Z namespaces/openshift-ingress/pods/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.053170145Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-w6tmc/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.053195296Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-w6tmc/istiod-openshift-gateway-5677ccb49b-w6tmc.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.053326178Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-w6tmc/discovery/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.053334409Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-w6tmc/discovery/discovery/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.053339499Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-w6tmc/discovery/discovery/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.05338311Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-w6tmc/discovery/discovery/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.063768381Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-w6tmc/discovery/discovery/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.063825082Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-w6tmc/discovery/discovery/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.063892473Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.063938344Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.064282881Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf/istio-proxy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.064294741Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf/istio-proxy/istio-proxy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.064300021Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.064333622Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.06572588Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.065815282Z namespaces/openshift-ingress/pods/router-default-65b54959b5-rhznz/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.065845232Z namespaces/openshift-ingress/pods/router-default-65b54959b5-rhznz/router-default-65b54959b5-rhznz.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.065981185Z namespaces/openshift-ingress/pods/router-default-65b54959b5-rhznz/router/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.065988466Z namespaces/openshift-ingress/pods/router-default-65b54959b5-rhznz/router/router/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.065997826Z namespaces/openshift-ingress/pods/router-default-65b54959b5-rhznz/router/router/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.066051897Z namespaces/openshift-ingress/pods/router-default-65b54959b5-rhznz/router/router/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.066165609Z namespaces/openshift-ingress/pods/router-default-65b54959b5-rhznz/router/router/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.06623024Z namespaces/openshift-ingress/pods/router-default-65b54959b5-rhznz/router/router/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.066278161Z namespaces/openshift-ingress/policy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.066294272Z namespaces/openshift-ingress/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.066368383Z namespaces/openshift-ingress/route.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.066414434Z namespaces/openshift-ingress/route.openshift.io/routes.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.066510626Z namespaces/openshift-insights/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.066545467Z namespaces/openshift-insights/openshift-insights.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.066658429Z namespaces/openshift-insights/apps.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.066680169Z namespaces/openshift-insights/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.066765771Z namespaces/openshift-insights/apps/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.066804622Z namespaces/openshift-insights/apps/daemonsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.066924985Z namespaces/openshift-insights/apps/deployments.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.067035747Z namespaces/openshift-insights/apps/replicasets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.067139929Z namespaces/openshift-insights/apps/statefulsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.06718528Z namespaces/openshift-insights/apps/deployments/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.067220411Z namespaces/openshift-insights/apps/deployments/insights-operator.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.067297102Z namespaces/openshift-insights/autoscaling/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.067344003Z namespaces/openshift-insights/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.067402304Z namespaces/openshift-insights/batch/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.067441495Z namespaces/openshift-insights/batch/cronjobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.067532807Z namespaces/openshift-insights/batch/jobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.067640339Z namespaces/openshift-insights/build.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.06769486Z namespaces/openshift-insights/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.067793782Z namespaces/openshift-insights/build.openshift.io/builds.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.067866213Z namespaces/openshift-insights/core/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.067915765Z namespaces/openshift-insights/core/configmaps.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.068895024Z namespaces/openshift-insights/core/endpoints.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.068987326Z namespaces/openshift-insights/core/events.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.069275322Z namespaces/openshift-insights/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.069402625Z namespaces/openshift-insights/core/pods.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.069748722Z namespaces/openshift-insights/core/replicationcontrollers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.069835473Z namespaces/openshift-insights/core/secrets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.069991257Z namespaces/openshift-insights/core/services.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.070059968Z namespaces/openshift-insights/core/configmaps/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.070088219Z namespaces/openshift-insights/core/configmaps/service-ca-bundle.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.07016306Z namespaces/openshift-insights/core/serviceaccounts/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.070208001Z namespaces/openshift-insights/core/serviceaccounts/gather.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.070294613Z namespaces/openshift-insights/core/serviceaccounts/operator.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.070349154Z namespaces/openshift-insights/core/services/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.070380555Z namespaces/openshift-insights/core/services/metrics.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.070455736Z namespaces/openshift-insights/discovery.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.070487977Z namespaces/openshift-insights/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.070590919Z namespaces/openshift-insights/image.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.07062548Z namespaces/openshift-insights/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.070685921Z namespaces/openshift-insights/k8s.ovn.org/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.070742462Z namespaces/openshift-insights/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.070824493Z namespaces/openshift-insights/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.070904465Z namespaces/openshift-insights/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.070960926Z namespaces/openshift-insights/monitoring.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.070990017Z namespaces/openshift-insights/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.071066189Z namespaces/openshift-insights/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.071094329Z namespaces/openshift-insights/monitoring.coreos.com/prometheusrules/insights-prometheus-rules.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.071174321Z namespaces/openshift-insights/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.071202921Z namespaces/openshift-insights/monitoring.coreos.com/servicemonitors/insights-operator.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.071280623Z namespaces/openshift-insights/networking.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.071314414Z namespaces/openshift-insights/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.071362975Z namespaces/openshift-insights/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.071372955Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.071426486Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.0716498Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.071868435Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.072012368Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.072220062Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.072461207Z namespaces/openshift-insights/pods/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.072469937Z namespaces/openshift-insights/pods/insights-operator-5885fdb759-r6ptx/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.072505828Z namespaces/openshift-insights/pods/insights-operator-5885fdb759-r6ptx/insights-operator-5885fdb759-r6ptx.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.07263124Z namespaces/openshift-insights/pods/insights-operator-5885fdb759-r6ptx/insights-operator/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.07264275Z namespaces/openshift-insights/pods/insights-operator-5885fdb759-r6ptx/insights-operator/insights-operator/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.072652851Z namespaces/openshift-insights/pods/insights-operator-5885fdb759-r6ptx/insights-operator/insights-operator/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.072698842Z namespaces/openshift-insights/pods/insights-operator-5885fdb759-r6ptx/insights-operator/insights-operator/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.07309759Z namespaces/openshift-insights/pods/insights-operator-5885fdb759-r6ptx/insights-operator/insights-operator/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.073414906Z namespaces/openshift-insights/pods/insights-runtime-extractor-mpcmc/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.073455747Z namespaces/openshift-insights/pods/insights-runtime-extractor-mpcmc/insights-runtime-extractor-mpcmc.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.073552389Z namespaces/openshift-insights/pods/insights-runtime-extractor-mpcmc/exporter/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.073561879Z namespaces/openshift-insights/pods/insights-runtime-extractor-mpcmc/exporter/exporter/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.073567579Z namespaces/openshift-insights/pods/insights-runtime-extractor-mpcmc/exporter/exporter/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.07360414Z namespaces/openshift-insights/pods/insights-runtime-extractor-mpcmc/exporter/exporter/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.073714602Z namespaces/openshift-insights/pods/insights-runtime-extractor-mpcmc/exporter/exporter/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.073780033Z namespaces/openshift-insights/pods/insights-runtime-extractor-mpcmc/exporter/exporter/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.073798314Z namespaces/openshift-insights/pods/insights-runtime-extractor-mpcmc/extractor/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.073803424Z namespaces/openshift-insights/pods/insights-runtime-extractor-mpcmc/extractor/extractor/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.073810814Z namespaces/openshift-insights/pods/insights-runtime-extractor-mpcmc/extractor/extractor/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.073858365Z namespaces/openshift-insights/pods/insights-runtime-extractor-mpcmc/extractor/extractor/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.073962377Z namespaces/openshift-insights/pods/insights-runtime-extractor-mpcmc/extractor/extractor/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.074028418Z namespaces/openshift-insights/pods/insights-runtime-extractor-mpcmc/extractor/extractor/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.074044969Z namespaces/openshift-insights/pods/insights-runtime-extractor-mpcmc/kube-rbac-proxy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.074054159Z namespaces/openshift-insights/pods/insights-runtime-extractor-mpcmc/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.074063729Z namespaces/openshift-insights/pods/insights-runtime-extractor-mpcmc/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.07412896Z namespaces/openshift-insights/pods/insights-runtime-extractor-mpcmc/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.074213312Z namespaces/openshift-insights/pods/insights-runtime-extractor-mpcmc/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.074289894Z namespaces/openshift-insights/pods/insights-runtime-extractor-mpcmc/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.074329585Z namespaces/openshift-insights/pods/insights-runtime-extractor-mtcd6/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.074357125Z namespaces/openshift-insights/pods/insights-runtime-extractor-mtcd6/insights-runtime-extractor-mtcd6.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.074455047Z namespaces/openshift-insights/pods/insights-runtime-extractor-mtcd6/exporter/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.074462117Z namespaces/openshift-insights/pods/insights-runtime-extractor-mtcd6/exporter/exporter/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.074473618Z namespaces/openshift-insights/pods/insights-runtime-extractor-mtcd6/exporter/exporter/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.074525919Z namespaces/openshift-insights/pods/insights-runtime-extractor-mtcd6/exporter/exporter/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.074657121Z namespaces/openshift-insights/pods/insights-runtime-extractor-mtcd6/exporter/exporter/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.074720912Z namespaces/openshift-insights/pods/insights-runtime-extractor-mtcd6/exporter/exporter/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.074756463Z namespaces/openshift-insights/pods/insights-runtime-extractor-mtcd6/extractor/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.074770073Z namespaces/openshift-insights/pods/insights-runtime-extractor-mtcd6/extractor/extractor/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.074775804Z namespaces/openshift-insights/pods/insights-runtime-extractor-mtcd6/extractor/extractor/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.074819765Z namespaces/openshift-insights/pods/insights-runtime-extractor-mtcd6/extractor/extractor/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.074917276Z namespaces/openshift-insights/pods/insights-runtime-extractor-mtcd6/extractor/extractor/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.074983238Z namespaces/openshift-insights/pods/insights-runtime-extractor-mtcd6/extractor/extractor/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.075002818Z namespaces/openshift-insights/pods/insights-runtime-extractor-mtcd6/kube-rbac-proxy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.075039639Z namespaces/openshift-insights/pods/insights-runtime-extractor-mtcd6/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.075048159Z namespaces/openshift-insights/pods/insights-runtime-extractor-mtcd6/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.07507526Z namespaces/openshift-insights/pods/insights-runtime-extractor-mtcd6/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.075161081Z namespaces/openshift-insights/pods/insights-runtime-extractor-mtcd6/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.075224953Z namespaces/openshift-insights/pods/insights-runtime-extractor-mtcd6/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.075253543Z namespaces/openshift-insights/pods/insights-runtime-extractor-vj25w/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.075296544Z namespaces/openshift-insights/pods/insights-runtime-extractor-vj25w/insights-runtime-extractor-vj25w.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.075394886Z namespaces/openshift-insights/pods/insights-runtime-extractor-vj25w/exporter/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.075403517Z namespaces/openshift-insights/pods/insights-runtime-extractor-vj25w/exporter/exporter/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.075408526Z namespaces/openshift-insights/pods/insights-runtime-extractor-vj25w/exporter/exporter/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.075451907Z namespaces/openshift-insights/pods/insights-runtime-extractor-vj25w/exporter/exporter/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.075535119Z namespaces/openshift-insights/pods/insights-runtime-extractor-vj25w/exporter/exporter/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.075636471Z namespaces/openshift-insights/pods/insights-runtime-extractor-vj25w/exporter/exporter/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.075659042Z namespaces/openshift-insights/pods/insights-runtime-extractor-vj25w/extractor/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.075668862Z namespaces/openshift-insights/pods/insights-runtime-extractor-vj25w/extractor/extractor/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.075673662Z namespaces/openshift-insights/pods/insights-runtime-extractor-vj25w/extractor/extractor/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.075720263Z namespaces/openshift-insights/pods/insights-runtime-extractor-vj25w/extractor/extractor/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.075827395Z namespaces/openshift-insights/pods/insights-runtime-extractor-vj25w/extractor/extractor/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.075891506Z namespaces/openshift-insights/pods/insights-runtime-extractor-vj25w/extractor/extractor/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.075933807Z namespaces/openshift-insights/pods/insights-runtime-extractor-vj25w/kube-rbac-proxy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.075940617Z namespaces/openshift-insights/pods/insights-runtime-extractor-vj25w/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.075944608Z namespaces/openshift-insights/pods/insights-runtime-extractor-vj25w/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.075965938Z namespaces/openshift-insights/pods/insights-runtime-extractor-vj25w/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.07607953Z namespaces/openshift-insights/pods/insights-runtime-extractor-vj25w/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.076147701Z namespaces/openshift-insights/pods/insights-runtime-extractor-vj25w/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.076171922Z namespaces/openshift-insights/policy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.076224713Z namespaces/openshift-insights/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.076276364Z namespaces/openshift-insights/route.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.076314475Z namespaces/openshift-insights/route.openshift.io/routes.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.076368266Z namespaces/openshift-keda/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.076375486Z namespaces/openshift-keda/coordination.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.076379466Z namespaces/openshift-keda/coordination.k8s.io/leases/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.076414467Z namespaces/openshift-keda/coordination.k8s.io/leases/olm-operator.keda.sh.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.076506459Z namespaces/openshift-keda/coordination.k8s.io/leases/operator.keda.sh.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.07655339Z namespaces/openshift-keda/monitoring.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.07656123Z namespaces/openshift-keda/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.076663352Z namespaces/openshift-keda/monitoring.coreos.com/podmonitors/keda-olm-operator.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.076698113Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.076749604Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/keda-admission-webhooks.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.076829735Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/keda-metrics-apiserver.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.076916927Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/keda-operator.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.076960898Z namespaces/openshift-keda/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.076970518Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.077018189Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.077169232Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.077360666Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/custom-metrics-autoscaler.v2.19.0-2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.07754987Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.077784645Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.077934968Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.078107881Z namespaces/openshift-keda/operators.coreos.com/installplans/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.078142132Z namespaces/openshift-keda/operators.coreos.com/installplans/install-rj6xp.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.078275175Z namespaces/openshift-keda/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.078309496Z namespaces/openshift-keda/operators.coreos.com/operatorconditions/custom-metrics-autoscaler.v2.19.0-2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.078345576Z namespaces/openshift-keda/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.078402027Z namespaces/openshift-keda/operators.coreos.com/operatorgroups/openshift-keda.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.078437068Z namespaces/openshift-keda/operators.coreos.com/subscriptions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.078488349Z namespaces/openshift-keda/operators.coreos.com/subscriptions/openshift-custom-metrics-autoscaler-operator.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.078591641Z namespaces/openshift-kube-apiserver-operator/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.078659792Z namespaces/openshift-kube-apiserver-operator/openshift-kube-apiserver-operator.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.078723214Z namespaces/openshift-kube-apiserver-operator/apps.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.078757834Z namespaces/openshift-kube-apiserver-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.078814526Z namespaces/openshift-kube-apiserver-operator/apps/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.078849116Z namespaces/openshift-kube-apiserver-operator/apps/daemonsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.078933038Z namespaces/openshift-kube-apiserver-operator/apps/deployments.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.079007819Z namespaces/openshift-kube-apiserver-operator/apps/replicasets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.079089221Z namespaces/openshift-kube-apiserver-operator/apps/statefulsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.079132522Z namespaces/openshift-kube-apiserver-operator/autoscaling/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.079163183Z namespaces/openshift-kube-apiserver-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.079217484Z namespaces/openshift-kube-apiserver-operator/batch/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.079250395Z namespaces/openshift-kube-apiserver-operator/batch/cronjobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.079331666Z namespaces/openshift-kube-apiserver-operator/batch/jobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.079381067Z namespaces/openshift-kube-apiserver-operator/build.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.079425718Z namespaces/openshift-kube-apiserver-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.079499229Z namespaces/openshift-kube-apiserver-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.079554901Z namespaces/openshift-kube-apiserver-operator/core/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.079622842Z namespaces/openshift-kube-apiserver-operator/core/configmaps.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.079729944Z namespaces/openshift-kube-apiserver-operator/core/endpoints.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.079805946Z namespaces/openshift-kube-apiserver-operator/core/events.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.079886847Z namespaces/openshift-kube-apiserver-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.079960149Z namespaces/openshift-kube-apiserver-operator/core/pods.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.0800406Z namespaces/openshift-kube-apiserver-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.080170483Z namespaces/openshift-kube-apiserver-operator/core/secrets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.080347037Z namespaces/openshift-kube-apiserver-operator/core/services.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.080399738Z namespaces/openshift-kube-apiserver-operator/discovery.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.080442479Z namespaces/openshift-kube-apiserver-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.08049135Z namespaces/openshift-kube-apiserver-operator/image.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.080537831Z namespaces/openshift-kube-apiserver-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.080589342Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.080658603Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.080735045Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.080822187Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.080850687Z namespaces/openshift-kube-apiserver-operator/monitoring.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.080895798Z namespaces/openshift-kube-apiserver-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.080950599Z namespaces/openshift-kube-apiserver-operator/networking.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.080977979Z namespaces/openshift-kube-apiserver-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.081028631Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.081035671Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.081078512Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.081240425Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.081416128Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.081550981Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.081777516Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.08197464Z namespaces/openshift-kube-apiserver-operator/policy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.082016Z namespaces/openshift-kube-apiserver-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.082068002Z namespaces/openshift-kube-apiserver-operator/route.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.082098952Z namespaces/openshift-kube-apiserver-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.082170734Z namespaces/openshift-kube-apiserver/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.082214885Z namespaces/openshift-kube-apiserver/openshift-kube-apiserver.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.082275076Z namespaces/openshift-kube-apiserver/apps.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.082310517Z namespaces/openshift-kube-apiserver/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.082346517Z namespaces/openshift-kube-apiserver/apps/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.082402539Z namespaces/openshift-kube-apiserver/apps/daemonsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.08247384Z namespaces/openshift-kube-apiserver/apps/deployments.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.082564642Z namespaces/openshift-kube-apiserver/apps/replicasets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.082669484Z namespaces/openshift-kube-apiserver/apps/statefulsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.082718015Z namespaces/openshift-kube-apiserver/autoscaling/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.082755226Z namespaces/openshift-kube-apiserver/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.082817577Z namespaces/openshift-kube-apiserver/batch/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.082852348Z namespaces/openshift-kube-apiserver/batch/cronjobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.082935479Z namespaces/openshift-kube-apiserver/batch/jobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.08299027Z namespaces/openshift-kube-apiserver/build.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.083031901Z namespaces/openshift-kube-apiserver/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.083123813Z namespaces/openshift-kube-apiserver/build.openshift.io/builds.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.083218015Z namespaces/openshift-kube-apiserver/core/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.083263516Z namespaces/openshift-kube-apiserver/core/configmaps.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.083363188Z namespaces/openshift-kube-apiserver/core/endpoints.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.083436949Z namespaces/openshift-kube-apiserver/core/events.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.083517431Z namespaces/openshift-kube-apiserver/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.083602913Z namespaces/openshift-kube-apiserver/core/pods.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.083735486Z namespaces/openshift-kube-apiserver/core/replicationcontrollers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.083857198Z namespaces/openshift-kube-apiserver/core/secrets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.084028872Z namespaces/openshift-kube-apiserver/core/services.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.084074302Z namespaces/openshift-kube-apiserver/discovery.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.084118823Z namespaces/openshift-kube-apiserver/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.084155914Z namespaces/openshift-kube-apiserver/image.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.084182875Z namespaces/openshift-kube-apiserver/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.084242046Z namespaces/openshift-kube-apiserver/k8s.ovn.org/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.084280086Z namespaces/openshift-kube-apiserver/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.084360258Z namespaces/openshift-kube-apiserver/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.08443406Z namespaces/openshift-kube-apiserver/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.084494151Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.084530872Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.084667094Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.084712105Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/prometheusrules/api-usage.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.084811207Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/prometheusrules/podsecurity.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.084892499Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.08492617Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/servicemonitors/openshift-kube-apiserver.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.084990521Z namespaces/openshift-kube-apiserver/networking.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.085031782Z namespaces/openshift-kube-apiserver/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.085079933Z namespaces/openshift-kube-apiserver/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.085087023Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.085128834Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.085272727Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.085627284Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.085763857Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.085974281Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.086164935Z namespaces/openshift-kube-apiserver/policy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.086205976Z namespaces/openshift-kube-apiserver/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.086257957Z namespaces/openshift-kube-apiserver/route.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.086290457Z namespaces/openshift-kube-apiserver/route.openshift.io/routes.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.086363289Z namespaces/openshift-kube-controller-manager-operator/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.08641562Z namespaces/openshift-kube-controller-manager-operator/openshift-kube-controller-manager-operator.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.086461461Z namespaces/openshift-kube-controller-manager-operator/apps.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.086502872Z namespaces/openshift-kube-controller-manager-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.086590993Z namespaces/openshift-kube-controller-manager-operator/apps/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.086627764Z namespaces/openshift-kube-controller-manager-operator/apps/daemonsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.086715016Z namespaces/openshift-kube-controller-manager-operator/apps/deployments.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.086788588Z namespaces/openshift-kube-controller-manager-operator/apps/replicasets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.086868649Z namespaces/openshift-kube-controller-manager-operator/apps/statefulsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.0869024Z namespaces/openshift-kube-controller-manager-operator/autoscaling/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.086934411Z namespaces/openshift-kube-controller-manager-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.086996472Z namespaces/openshift-kube-controller-manager-operator/batch/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.087023582Z namespaces/openshift-kube-controller-manager-operator/batch/cronjobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.087108594Z namespaces/openshift-kube-controller-manager-operator/batch/jobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.087157075Z namespaces/openshift-kube-controller-manager-operator/build.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.087196596Z namespaces/openshift-kube-controller-manager-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.087348069Z namespaces/openshift-kube-controller-manager-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.08740536Z namespaces/openshift-kube-controller-manager-operator/core/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.087453711Z namespaces/openshift-kube-controller-manager-operator/core/configmaps.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.087560873Z namespaces/openshift-kube-controller-manager-operator/core/endpoints.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.087661755Z namespaces/openshift-kube-controller-manager-operator/core/events.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.087736907Z namespaces/openshift-kube-controller-manager-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.087818278Z namespaces/openshift-kube-controller-manager-operator/core/pods.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.08789072Z namespaces/openshift-kube-controller-manager-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.088010412Z namespaces/openshift-kube-controller-manager-operator/core/secrets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.088179916Z namespaces/openshift-kube-controller-manager-operator/core/services.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.088240777Z namespaces/openshift-kube-controller-manager-operator/discovery.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.088278758Z namespaces/openshift-kube-controller-manager-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.088332269Z namespaces/openshift-kube-controller-manager-operator/image.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.088358869Z namespaces/openshift-kube-controller-manager-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.088417341Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.088462961Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.088536913Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.088636675Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.088685636Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.088729627Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.088784478Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.088820099Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/servicemonitors/kube-controller-manager-operator.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.08888135Z namespaces/openshift-kube-controller-manager-operator/networking.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.088915751Z namespaces/openshift-kube-controller-manager-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.088963801Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.088973322Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.089014833Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.089167996Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.089347599Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.089483572Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.089697267Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.08988117Z namespaces/openshift-kube-controller-manager-operator/policy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.089919711Z namespaces/openshift-kube-controller-manager-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.089970632Z namespaces/openshift-kube-controller-manager-operator/route.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.090005293Z namespaces/openshift-kube-controller-manager-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.090072504Z namespaces/openshift-kube-controller-manager/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.090120715Z namespaces/openshift-kube-controller-manager/openshift-kube-controller-manager.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.090180516Z namespaces/openshift-kube-controller-manager/apps.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.090223387Z namespaces/openshift-kube-controller-manager/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.090277938Z namespaces/openshift-kube-controller-manager/apps/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.090313799Z namespaces/openshift-kube-controller-manager/apps/daemonsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.090395841Z namespaces/openshift-kube-controller-manager/apps/deployments.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.090472102Z namespaces/openshift-kube-controller-manager/apps/replicasets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.090596535Z namespaces/openshift-kube-controller-manager/apps/statefulsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.090649696Z namespaces/openshift-kube-controller-manager/autoscaling/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.090692747Z namespaces/openshift-kube-controller-manager/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.090745608Z namespaces/openshift-kube-controller-manager/batch/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.090777828Z namespaces/openshift-kube-controller-manager/batch/cronjobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.09085831Z namespaces/openshift-kube-controller-manager/batch/jobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.090913371Z namespaces/openshift-kube-controller-manager/build.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.090946022Z namespaces/openshift-kube-controller-manager/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.091032323Z namespaces/openshift-kube-controller-manager/build.openshift.io/builds.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.091077215Z namespaces/openshift-kube-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.091084265Z namespaces/openshift-kube-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.091122625Z namespaces/openshift-kube-controller-manager/coordination.k8s.io/leases/cluster-policy-controller-lock.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.091187407Z namespaces/openshift-kube-controller-manager/core/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.091223958Z namespaces/openshift-kube-controller-manager/core/configmaps.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.0913314Z namespaces/openshift-kube-controller-manager/core/endpoints.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.091406161Z namespaces/openshift-kube-controller-manager/core/events.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.091915601Z namespaces/openshift-kube-controller-manager/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.091990563Z namespaces/openshift-kube-controller-manager/core/pods.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.092072575Z namespaces/openshift-kube-controller-manager/core/replicationcontrollers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.092190037Z namespaces/openshift-kube-controller-manager/core/secrets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.09234215Z namespaces/openshift-kube-controller-manager/core/services.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.092388391Z namespaces/openshift-kube-controller-manager/discovery.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.092430762Z namespaces/openshift-kube-controller-manager/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.092470683Z namespaces/openshift-kube-controller-manager/image.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.092512624Z namespaces/openshift-kube-controller-manager/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.092589915Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.092675107Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.092765349Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.09284581Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.092892101Z namespaces/openshift-kube-controller-manager/monitoring.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.092930772Z namespaces/openshift-kube-controller-manager/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.092974573Z namespaces/openshift-kube-controller-manager/networking.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.093019564Z namespaces/openshift-kube-controller-manager/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.093066045Z namespaces/openshift-kube-controller-manager/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.093075465Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.093111336Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.093247988Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.093440452Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.093591266Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.093782739Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.093962713Z namespaces/openshift-kube-controller-manager/policy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.093986943Z namespaces/openshift-kube-controller-manager/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.094044445Z namespaces/openshift-kube-controller-manager/route.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.094084425Z namespaces/openshift-kube-controller-manager/route.openshift.io/routes.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.094141017Z namespaces/openshift-kube-scheduler-operator/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.094190148Z namespaces/openshift-kube-scheduler-operator/openshift-kube-scheduler-operator.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.094257379Z namespaces/openshift-kube-scheduler-operator/apps.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.0942816Z namespaces/openshift-kube-scheduler-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.094344081Z namespaces/openshift-kube-scheduler-operator/apps/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.094381462Z namespaces/openshift-kube-scheduler-operator/apps/daemonsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.094464313Z namespaces/openshift-kube-scheduler-operator/apps/deployments.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.094544435Z namespaces/openshift-kube-scheduler-operator/apps/replicasets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.094659717Z namespaces/openshift-kube-scheduler-operator/apps/statefulsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.094702218Z namespaces/openshift-kube-scheduler-operator/autoscaling/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.094744249Z namespaces/openshift-kube-scheduler-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.09479065Z namespaces/openshift-kube-scheduler-operator/batch/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.094829231Z namespaces/openshift-kube-scheduler-operator/batch/cronjobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.094911702Z namespaces/openshift-kube-scheduler-operator/batch/jobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.094964283Z namespaces/openshift-kube-scheduler-operator/build.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.095013584Z namespaces/openshift-kube-scheduler-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.095093996Z namespaces/openshift-kube-scheduler-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.095154977Z namespaces/openshift-kube-scheduler-operator/core/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.095189308Z namespaces/openshift-kube-scheduler-operator/core/configmaps.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.09529897Z namespaces/openshift-kube-scheduler-operator/core/endpoints.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.095383002Z namespaces/openshift-kube-scheduler-operator/core/events.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.095464133Z namespaces/openshift-kube-scheduler-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.095537345Z namespaces/openshift-kube-scheduler-operator/core/pods.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.095637147Z namespaces/openshift-kube-scheduler-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.09575865Z namespaces/openshift-kube-scheduler-operator/core/secrets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.095909173Z namespaces/openshift-kube-scheduler-operator/core/services.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.095956754Z namespaces/openshift-kube-scheduler-operator/discovery.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.095990964Z namespaces/openshift-kube-scheduler-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.096043055Z namespaces/openshift-kube-scheduler-operator/image.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.096089226Z namespaces/openshift-kube-scheduler-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.096163348Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.096209679Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.0962853Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.096367072Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.096400272Z namespaces/openshift-kube-scheduler-operator/monitoring.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.096432193Z namespaces/openshift-kube-scheduler-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.096491204Z namespaces/openshift-kube-scheduler-operator/networking.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.096518185Z namespaces/openshift-kube-scheduler-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.096586496Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.096597376Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.096637437Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.09678828Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.097037925Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.097172408Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.097357842Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.097552926Z namespaces/openshift-kube-scheduler-operator/policy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.097600947Z namespaces/openshift-kube-scheduler-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.097668648Z namespaces/openshift-kube-scheduler-operator/route.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.097704369Z namespaces/openshift-kube-scheduler-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.09777116Z namespaces/openshift-kube-scheduler/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.097817171Z namespaces/openshift-kube-scheduler/openshift-kube-scheduler.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.097875382Z namespaces/openshift-kube-scheduler/apps.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.097904183Z namespaces/openshift-kube-scheduler/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.097954714Z namespaces/openshift-kube-scheduler/apps/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.097986235Z namespaces/openshift-kube-scheduler/apps/daemonsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.098064406Z namespaces/openshift-kube-scheduler/apps/deployments.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.098143858Z namespaces/openshift-kube-scheduler/apps/replicasets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.098213779Z namespaces/openshift-kube-scheduler/apps/statefulsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.09826357Z namespaces/openshift-kube-scheduler/autoscaling/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.098296611Z namespaces/openshift-kube-scheduler/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.098352292Z namespaces/openshift-kube-scheduler/batch/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.098391273Z namespaces/openshift-kube-scheduler/batch/cronjobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.098472665Z namespaces/openshift-kube-scheduler/batch/jobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.098506435Z namespaces/openshift-kube-scheduler/build.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.098544636Z namespaces/openshift-kube-scheduler/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.098662828Z namespaces/openshift-kube-scheduler/build.openshift.io/builds.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.098696439Z namespaces/openshift-kube-scheduler/core/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.09874745Z namespaces/openshift-kube-scheduler/core/configmaps.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.098864462Z namespaces/openshift-kube-scheduler/core/endpoints.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.098938174Z namespaces/openshift-kube-scheduler/core/events.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.099017716Z namespaces/openshift-kube-scheduler/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.099090397Z namespaces/openshift-kube-scheduler/core/pods.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.099168739Z namespaces/openshift-kube-scheduler/core/replicationcontrollers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.099287881Z namespaces/openshift-kube-scheduler/core/secrets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.099443814Z namespaces/openshift-kube-scheduler/core/services.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.099478785Z namespaces/openshift-kube-scheduler/discovery.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.099532756Z namespaces/openshift-kube-scheduler/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.099588547Z namespaces/openshift-kube-scheduler/image.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.099643028Z namespaces/openshift-kube-scheduler/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.09971244Z namespaces/openshift-kube-scheduler/k8s.ovn.org/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.099757351Z namespaces/openshift-kube-scheduler/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.099842942Z namespaces/openshift-kube-scheduler/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.099932964Z namespaces/openshift-kube-scheduler/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.099979555Z namespaces/openshift-kube-scheduler/monitoring.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.100016346Z namespaces/openshift-kube-scheduler/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.100062457Z namespaces/openshift-kube-scheduler/networking.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.100105548Z namespaces/openshift-kube-scheduler/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.100151339Z namespaces/openshift-kube-scheduler/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.100158609Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.100196829Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.100344412Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.100522096Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.100753841Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.100942805Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.101129178Z namespaces/openshift-kube-scheduler/policy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.101157899Z namespaces/openshift-kube-scheduler/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.10121058Z namespaces/openshift-kube-scheduler/route.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.101254521Z namespaces/openshift-kube-scheduler/route.openshift.io/routes.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.101321232Z namespaces/openshift-kube-storage-version-migrator-operator/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.101375403Z namespaces/openshift-kube-storage-version-migrator-operator/openshift-kube-storage-version-migrator-operator.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.101428565Z namespaces/openshift-kube-storage-version-migrator-operator/apps.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.101471005Z namespaces/openshift-kube-storage-version-migrator-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.101534177Z namespaces/openshift-kube-storage-version-migrator-operator/apps/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.101593658Z namespaces/openshift-kube-storage-version-migrator-operator/apps/daemonsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.101676919Z namespaces/openshift-kube-storage-version-migrator-operator/apps/deployments.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.101789612Z namespaces/openshift-kube-storage-version-migrator-operator/apps/replicasets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.101904544Z namespaces/openshift-kube-storage-version-migrator-operator/apps/statefulsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.101957225Z namespaces/openshift-kube-storage-version-migrator-operator/autoscaling/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.101997246Z namespaces/openshift-kube-storage-version-migrator-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.102055557Z namespaces/openshift-kube-storage-version-migrator-operator/batch/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.102091558Z namespaces/openshift-kube-storage-version-migrator-operator/batch/cronjobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.10217678Z namespaces/openshift-kube-storage-version-migrator-operator/batch/jobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.102235431Z namespaces/openshift-kube-storage-version-migrator-operator/build.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.102263701Z namespaces/openshift-kube-storage-version-migrator-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.102355123Z namespaces/openshift-kube-storage-version-migrator-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.102397874Z namespaces/openshift-kube-storage-version-migrator-operator/coordination.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.102407314Z namespaces/openshift-kube-storage-version-migrator-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.102456105Z namespaces/openshift-kube-storage-version-migrator-operator/coordination.k8s.io/leases/openshift-kube-storage-version-migrator-operator-lock.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.102547847Z namespaces/openshift-kube-storage-version-migrator-operator/core/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.102595428Z namespaces/openshift-kube-storage-version-migrator-operator/core/configmaps.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.102721721Z namespaces/openshift-kube-storage-version-migrator-operator/core/endpoints.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.102811513Z namespaces/openshift-kube-storage-version-migrator-operator/core/events.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.102980446Z namespaces/openshift-kube-storage-version-migrator-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.103053618Z namespaces/openshift-kube-storage-version-migrator-operator/core/pods.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.103283082Z namespaces/openshift-kube-storage-version-migrator-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.103401734Z namespaces/openshift-kube-storage-version-migrator-operator/core/secrets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.103588668Z namespaces/openshift-kube-storage-version-migrator-operator/core/services.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.10365331Z namespaces/openshift-kube-storage-version-migrator-operator/discovery.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.103699331Z namespaces/openshift-kube-storage-version-migrator-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.103752862Z namespaces/openshift-kube-storage-version-migrator-operator/image.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.103793962Z namespaces/openshift-kube-storage-version-migrator-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.103851154Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.103893855Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.103979336Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.104053518Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.104112579Z namespaces/openshift-kube-storage-version-migrator-operator/monitoring.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.1041367Z namespaces/openshift-kube-storage-version-migrator-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.104205081Z namespaces/openshift-kube-storage-version-migrator-operator/networking.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.104247792Z namespaces/openshift-kube-storage-version-migrator-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.104312063Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.104319463Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.104356064Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.104512247Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.104719241Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.104870474Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.105066978Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.105274872Z namespaces/openshift-kube-storage-version-migrator-operator/pods/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.105283603Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6f4d6fb6bc-czhcl/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.105326204Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6f4d6fb6bc-czhcl/kube-storage-version-migrator-operator-6f4d6fb6bc-czhcl.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.105445176Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6f4d6fb6bc-czhcl/kube-storage-version-migrator-operator/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.105452916Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6f4d6fb6bc-czhcl/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.105461566Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6f4d6fb6bc-czhcl/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.105512837Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6f4d6fb6bc-czhcl/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.10565703Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6f4d6fb6bc-czhcl/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.105776063Z namespaces/openshift-kube-storage-version-migrator-operator/policy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.105793033Z namespaces/openshift-kube-storage-version-migrator-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.105855274Z namespaces/openshift-kube-storage-version-migrator-operator/route.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.105896375Z namespaces/openshift-kube-storage-version-migrator-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.105954716Z namespaces/openshift-kube-storage-version-migrator/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.106007547Z namespaces/openshift-kube-storage-version-migrator/openshift-kube-storage-version-migrator.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.106102909Z namespaces/openshift-kube-storage-version-migrator/apps.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.10614255Z namespaces/openshift-kube-storage-version-migrator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.106197031Z namespaces/openshift-kube-storage-version-migrator/apps/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.106225662Z namespaces/openshift-kube-storage-version-migrator/apps/daemonsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.106319564Z namespaces/openshift-kube-storage-version-migrator/apps/deployments.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.106427116Z namespaces/openshift-kube-storage-version-migrator/apps/replicasets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.106522498Z namespaces/openshift-kube-storage-version-migrator/apps/statefulsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.10660667Z namespaces/openshift-kube-storage-version-migrator/autoscaling/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.10664142Z namespaces/openshift-kube-storage-version-migrator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.106692551Z namespaces/openshift-kube-storage-version-migrator/batch/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.106731552Z namespaces/openshift-kube-storage-version-migrator/batch/cronjobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.106819724Z namespaces/openshift-kube-storage-version-migrator/batch/jobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.106869075Z namespaces/openshift-kube-storage-version-migrator/build.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.106887185Z namespaces/openshift-kube-storage-version-migrator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.106994917Z namespaces/openshift-kube-storage-version-migrator/build.openshift.io/builds.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.107014278Z namespaces/openshift-kube-storage-version-migrator/core/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.107065199Z namespaces/openshift-kube-storage-version-migrator/core/configmaps.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.107164571Z namespaces/openshift-kube-storage-version-migrator/core/endpoints.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.107239322Z namespaces/openshift-kube-storage-version-migrator/core/events.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.107348985Z namespaces/openshift-kube-storage-version-migrator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.107429606Z namespaces/openshift-kube-storage-version-migrator/core/pods.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.107545109Z namespaces/openshift-kube-storage-version-migrator/core/replicationcontrollers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.107678271Z namespaces/openshift-kube-storage-version-migrator/core/secrets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.107834555Z namespaces/openshift-kube-storage-version-migrator/core/services.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.107869875Z namespaces/openshift-kube-storage-version-migrator/discovery.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.107914596Z namespaces/openshift-kube-storage-version-migrator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.107971957Z namespaces/openshift-kube-storage-version-migrator/image.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.107994038Z namespaces/openshift-kube-storage-version-migrator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.108056269Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.1080941Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.108173031Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.108255453Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.108296314Z namespaces/openshift-kube-storage-version-migrator/monitoring.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.108328794Z namespaces/openshift-kube-storage-version-migrator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.108366795Z namespaces/openshift-kube-storage-version-migrator/networking.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.108415246Z namespaces/openshift-kube-storage-version-migrator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.108462277Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.108471277Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.108516828Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.108683372Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.108861055Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.108991508Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.109357505Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.109553049Z namespaces/openshift-kube-storage-version-migrator/pods/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.109563799Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-wdgll/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.109684932Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-wdgll/migrator-6865c9967-wdgll.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.109762794Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-wdgll/graceful-termination/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.109775184Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-wdgll/graceful-termination/graceful-termination/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.109809195Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-wdgll/graceful-termination/graceful-termination/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.109859316Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-wdgll/graceful-termination/graceful-termination/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.109945687Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-wdgll/graceful-termination/graceful-termination/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.110019339Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-wdgll/graceful-termination/graceful-termination/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.110044989Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-wdgll/migrator/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.11005306Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-wdgll/migrator/migrator/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.11006361Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-wdgll/migrator/migrator/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.11010273Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-wdgll/migrator/migrator/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.110200872Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-wdgll/migrator/migrator/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.110272844Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-wdgll/migrator/migrator/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.110284244Z namespaces/openshift-kube-storage-version-migrator/policy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.110335825Z namespaces/openshift-kube-storage-version-migrator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.110386736Z namespaces/openshift-kube-storage-version-migrator/route.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.110441367Z namespaces/openshift-kube-storage-version-migrator/route.openshift.io/routes.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.110508219Z namespaces/openshift-lws-operator/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.11056482Z namespaces/openshift-lws-operator/openshift-lws-operator.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.110629541Z namespaces/openshift-lws-operator/apps.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.110696993Z namespaces/openshift-lws-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.110750023Z namespaces/openshift-lws-operator/apps/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.110801975Z namespaces/openshift-lws-operator/apps/daemonsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.110874286Z namespaces/openshift-lws-operator/apps/deployments.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.111017769Z namespaces/openshift-lws-operator/apps/replicasets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.111135571Z namespaces/openshift-lws-operator/apps/statefulsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.111191503Z namespaces/openshift-lws-operator/autoscaling/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.111275104Z namespaces/openshift-lws-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.111316555Z namespaces/openshift-lws-operator/batch/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.111350346Z namespaces/openshift-lws-operator/batch/cronjobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.111439068Z namespaces/openshift-lws-operator/batch/jobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.111472818Z namespaces/openshift-lws-operator/build.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.111521869Z namespaces/openshift-lws-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.111632621Z namespaces/openshift-lws-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.111694523Z namespaces/openshift-lws-operator/coordination.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.111701763Z namespaces/openshift-lws-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.111736204Z namespaces/openshift-lws-operator/coordination.k8s.io/leases/b8b2488c.x-k8s.io.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.111828965Z namespaces/openshift-lws-operator/coordination.k8s.io/leases/openshift-lws-operator-lock.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.111881717Z namespaces/openshift-lws-operator/core/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.111922998Z namespaces/openshift-lws-operator/core/configmaps.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.11203331Z namespaces/openshift-lws-operator/core/endpoints.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.112117891Z namespaces/openshift-lws-operator/core/events.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.112431508Z namespaces/openshift-lws-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.112509479Z namespaces/openshift-lws-operator/core/pods.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.112727184Z namespaces/openshift-lws-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.112845176Z namespaces/openshift-lws-operator/core/secrets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.1130389Z namespaces/openshift-lws-operator/core/services.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.113096631Z namespaces/openshift-lws-operator/discovery.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.113144232Z namespaces/openshift-lws-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.113201073Z namespaces/openshift-lws-operator/image.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.113229664Z namespaces/openshift-lws-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.113290475Z namespaces/openshift-lws-operator/k8s.ovn.org/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.113318146Z namespaces/openshift-lws-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.113409408Z namespaces/openshift-lws-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.113490379Z namespaces/openshift-lws-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.11354563Z namespaces/openshift-lws-operator/monitoring.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.113601081Z namespaces/openshift-lws-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.113667333Z namespaces/openshift-lws-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.113700513Z namespaces/openshift-lws-operator/monitoring.coreos.com/servicemonitors/lws-controller-manager-metrics-monitor.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.113766125Z namespaces/openshift-lws-operator/networking.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.113803765Z namespaces/openshift-lws-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.113861837Z namespaces/openshift-lws-operator/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.113868717Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.113923018Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.114068771Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.114255285Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.114400358Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/leader-worker-set.v1.0.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.114654433Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.114936879Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.115121152Z namespaces/openshift-lws-operator/operators.coreos.com/installplans/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.115160603Z namespaces/openshift-lws-operator/operators.coreos.com/installplans/install-nr74g.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.115242065Z namespaces/openshift-lws-operator/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.115266995Z namespaces/openshift-lws-operator/operators.coreos.com/operatorconditions/leader-worker-set.v1.0.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.115326236Z namespaces/openshift-lws-operator/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.115370017Z namespaces/openshift-lws-operator/operators.coreos.com/operatorgroups/leader-worker-set.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.115419398Z namespaces/openshift-lws-operator/operators.coreos.com/subscriptions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.115459139Z namespaces/openshift-lws-operator/operators.coreos.com/subscriptions/leader-worker-set.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.115534931Z namespaces/openshift-lws-operator/pods/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.115542481Z namespaces/openshift-lws-operator/pods/lws-controller-manager-68cc75d5fc-8886b/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.115591032Z namespaces/openshift-lws-operator/pods/lws-controller-manager-68cc75d5fc-8886b/lws-controller-manager-68cc75d5fc-8886b.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.115700014Z namespaces/openshift-lws-operator/pods/lws-controller-manager-68cc75d5fc-8886b/manager/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.115716444Z namespaces/openshift-lws-operator/pods/lws-controller-manager-68cc75d5fc-8886b/manager/manager/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.115722115Z namespaces/openshift-lws-operator/pods/lws-controller-manager-68cc75d5fc-8886b/manager/manager/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.115749845Z namespaces/openshift-lws-operator/pods/lws-controller-manager-68cc75d5fc-8886b/manager/manager/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.115866657Z namespaces/openshift-lws-operator/pods/lws-controller-manager-68cc75d5fc-8886b/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.115932839Z namespaces/openshift-lws-operator/pods/lws-controller-manager-68cc75d5fc-8886b/manager/manager/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.11598876Z namespaces/openshift-lws-operator/pods/lws-controller-manager-68cc75d5fc-895fk/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.11601544Z namespaces/openshift-lws-operator/pods/lws-controller-manager-68cc75d5fc-895fk/lws-controller-manager-68cc75d5fc-895fk.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.116104042Z namespaces/openshift-lws-operator/pods/lws-controller-manager-68cc75d5fc-895fk/manager/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.116111742Z namespaces/openshift-lws-operator/pods/lws-controller-manager-68cc75d5fc-895fk/manager/manager/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.116115752Z namespaces/openshift-lws-operator/pods/lws-controller-manager-68cc75d5fc-895fk/manager/manager/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.116157333Z namespaces/openshift-lws-operator/pods/lws-controller-manager-68cc75d5fc-895fk/manager/manager/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.11647492Z namespaces/openshift-lws-operator/pods/lws-controller-manager-68cc75d5fc-895fk/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.116557021Z namespaces/openshift-lws-operator/pods/lws-controller-manager-68cc75d5fc-895fk/manager/manager/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.116629093Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-mzlfv/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.116669974Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-mzlfv/openshift-lws-operator-fd8ccff4c-mzlfv.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.116754225Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-mzlfv/openshift-lws-operator/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.116762996Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-mzlfv/openshift-lws-operator/openshift-lws-operator/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.116769936Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-mzlfv/openshift-lws-operator/openshift-lws-operator/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.116812497Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-mzlfv/openshift-lws-operator/openshift-lws-operator/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.11698228Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-mzlfv/openshift-lws-operator/openshift-lws-operator/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.117048731Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-mzlfv/openshift-lws-operator/openshift-lws-operator/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.117081402Z namespaces/openshift-lws-operator/policy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.117110813Z namespaces/openshift-lws-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.117169334Z namespaces/openshift-lws-operator/route.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.117201334Z namespaces/openshift-lws-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.117237395Z namespaces/openshift-machine-api/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.117250155Z namespaces/openshift-machine-api/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.117263906Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.117275696Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.11748327Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.117700285Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.117841147Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.118052912Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.118240946Z namespaces/openshift-machine-config-operator/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.118249526Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.118254646Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.118279886Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/prometheusrules/machine-config-controller.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.118402839Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/prometheusrules/machine-config-daemon.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.11847001Z namespaces/openshift-machine-config-operator/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.1184803Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.118522761Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.118791897Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.1189668Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.119106763Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.119341858Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.119533292Z namespaces/openshift-marketplace/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.119545832Z namespaces/openshift-marketplace/monitoring.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.119553482Z namespaces/openshift-marketplace/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.119565933Z namespaces/openshift-marketplace/monitoring.coreos.com/prometheusrules/marketplace-alert-rules.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.119661424Z namespaces/openshift-marketplace/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.119678695Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.119691015Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/certified-operators.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.119794127Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/community-operators.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.119872249Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/redhat-marketplace.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.11995407Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/redhat-operators.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.120005791Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.120052122Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.120200285Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.120375879Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.120508692Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.120721956Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.12091965Z namespaces/openshift-monitoring/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.12094937Z namespaces/openshift-monitoring/openshift-monitoring.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.121028802Z namespaces/openshift-monitoring/apps.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.121067403Z namespaces/openshift-monitoring/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.121133234Z namespaces/openshift-monitoring/apps/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.121174965Z namespaces/openshift-monitoring/apps/daemonsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.121313298Z namespaces/openshift-monitoring/apps/deployments.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.121755887Z namespaces/openshift-monitoring/apps/replicasets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.122231667Z namespaces/openshift-monitoring/apps/statefulsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.122464631Z namespaces/openshift-monitoring/autoscaling/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.122498172Z namespaces/openshift-monitoring/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.122554893Z namespaces/openshift-monitoring/batch/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.122597434Z namespaces/openshift-monitoring/batch/cronjobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.122718437Z namespaces/openshift-monitoring/batch/jobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.122798268Z namespaces/openshift-monitoring/build.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.122837009Z namespaces/openshift-monitoring/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.122920721Z namespaces/openshift-monitoring/build.openshift.io/builds.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.122971062Z namespaces/openshift-monitoring/core/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.122999102Z namespaces/openshift-monitoring/core/configmaps.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.126097655Z namespaces/openshift-monitoring/core/endpoints.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.126242378Z namespaces/openshift-monitoring/core/events.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.127370061Z namespaces/openshift-monitoring/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.127441472Z namespaces/openshift-monitoring/core/pods.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.128378511Z namespaces/openshift-monitoring/core/replicationcontrollers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.128498484Z namespaces/openshift-monitoring/core/secrets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.128989924Z namespaces/openshift-monitoring/core/services.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.129161317Z namespaces/openshift-monitoring/discovery.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.129183528Z namespaces/openshift-monitoring/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.129350811Z namespaces/openshift-monitoring/image.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.129371431Z namespaces/openshift-monitoring/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.129427953Z namespaces/openshift-monitoring/k8s.ovn.org/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.129471914Z namespaces/openshift-monitoring/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.129542785Z namespaces/openshift-monitoring/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.129642757Z namespaces/openshift-monitoring/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.129684548Z namespaces/openshift-monitoring/monitoring.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.129731649Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.129849531Z namespaces/openshift-monitoring/monitoring.coreos.com/alertmanagers/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.129889572Z namespaces/openshift-monitoring/monitoring.coreos.com/alertmanagers/main.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.129976474Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheuses/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.129993444Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheuses/k8s.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.130109726Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.130144547Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/alertmanager-main-rules.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.130246349Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/cluster-monitoring-operator-prometheus-rules.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.130426313Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/kube-state-metrics-rules.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.130504624Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/kubernetes-monitoring-rules.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.13077332Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/node-exporter-rules.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.130903232Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/prometheus-k8s-prometheus-rules.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.131027425Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/prometheus-k8s-thanos-sidecar-rules.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.131112137Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/prometheus-operator-rules.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.131214299Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/telemetry.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.13130248Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/thanos-querier.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.131377962Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.131409473Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/alertmanager-main.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.131515135Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/cluster-monitoring-operator.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.131617997Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/kube-state-metrics.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.131719669Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/kubelet.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.131817901Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/metrics-server.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.131900683Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/node-exporter.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.132056816Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/openshift-state-metrics.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.132137828Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/prometheus-k8s.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.13224046Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/prometheus-operator.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.132318031Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/telemeter-client.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.132403333Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/thanos-querier.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.132488645Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/thanos-sidecar.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.132561606Z namespaces/openshift-monitoring/networking.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.132615907Z namespaces/openshift-monitoring/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.132673458Z namespaces/openshift-monitoring/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.132680688Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.132719409Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.132900653Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.133138818Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.133281441Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.133478375Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.1337172Z namespaces/openshift-monitoring/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.13374765Z namespaces/openshift-monitoring/operators.coreos.com/operatorgroups/openshift-cluster-monitoring.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.133839302Z namespaces/openshift-monitoring/pods/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.133846472Z namespaces/openshift-monitoring/pods/alertmanager-main-0/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.133912324Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager-main-0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.134082937Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.134090907Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.134095707Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.134135068Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.13423036Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.134294991Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.134318072Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.134322592Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.134329072Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.134372583Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.134470355Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.134546637Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.134592647Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.134601818Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.134669929Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.13471669Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.134805922Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.134868223Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.134897733Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.134902364Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.134912474Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.134963075Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.135138218Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.13520221Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.1352399Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.135247641Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.135251611Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.135292521Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.135380533Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.135445565Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.135489525Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.135496896Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.135501146Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.135524076Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.135641719Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.13570892Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.13573969Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.135746941Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.135750821Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.135795102Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.135881013Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.135944005Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.135988606Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-x6bhh/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.136024786Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-x6bhh/cluster-monitoring-operator-548b489cb8-x6bhh.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.136117098Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-x6bhh/cluster-monitoring-operator/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.136124298Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-x6bhh/cluster-monitoring-operator/cluster-monitoring-operator/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.136128609Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-x6bhh/cluster-monitoring-operator/cluster-monitoring-operator/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.136168599Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-x6bhh/cluster-monitoring-operator/cluster-monitoring-operator/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.136920194Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-x6bhh/cluster-monitoring-operator/cluster-monitoring-operator/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.136987166Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-x6bhh/cluster-monitoring-operator/cluster-monitoring-operator/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.137021817Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-27zms/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.137056527Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-27zms/kube-state-metrics-56d9f944b4-27zms.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.1371799Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-27zms/kube-rbac-proxy-main/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.13718707Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-27zms/kube-rbac-proxy-main/kube-rbac-proxy-main/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.13719118Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-27zms/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.137236721Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-27zms/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.137324273Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-27zms/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.137390464Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-27zms/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.137424415Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-27zms/kube-rbac-proxy-self/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.137433165Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-27zms/kube-rbac-proxy-self/kube-rbac-proxy-self/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.137437145Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-27zms/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.137469756Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-27zms/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.137557807Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-27zms/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.1376583Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-27zms/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.13770437Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-27zms/kube-state-metrics/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.13770953Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-27zms/kube-state-metrics/kube-state-metrics/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.137715591Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-27zms/kube-state-metrics/kube-state-metrics/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.137757331Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-27zms/kube-state-metrics/kube-state-metrics/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.137983906Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-27zms/kube-state-metrics/kube-state-metrics/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.138052557Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-27zms/kube-state-metrics/kube-state-metrics/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.138091908Z namespaces/openshift-monitoring/pods/metrics-server-5b7c89b76c-w2zrh/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.138125779Z namespaces/openshift-monitoring/pods/metrics-server-5b7c89b76c-w2zrh/metrics-server-5b7c89b76c-w2zrh.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.138223071Z namespaces/openshift-monitoring/pods/metrics-server-5b7c89b76c-w2zrh/metrics-server/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.138230461Z namespaces/openshift-monitoring/pods/metrics-server-5b7c89b76c-w2zrh/metrics-server/metrics-server/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.138234771Z namespaces/openshift-monitoring/pods/metrics-server-5b7c89b76c-w2zrh/metrics-server/metrics-server/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.138263502Z namespaces/openshift-monitoring/pods/metrics-server-5b7c89b76c-w2zrh/metrics-server/metrics-server/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.138359844Z namespaces/openshift-monitoring/pods/metrics-server-5b7c89b76c-w2zrh/metrics-server/metrics-server/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.138426855Z namespaces/openshift-monitoring/pods/metrics-server-5b7c89b76c-w2zrh/metrics-server/metrics-server/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.138470566Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-blv5n/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.138498297Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-blv5n/monitoring-plugin-68b7fc9574-blv5n.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.138627699Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-blv5n/monitoring-plugin/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.138639729Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-blv5n/monitoring-plugin/monitoring-plugin/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.13864462Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-blv5n/monitoring-plugin/monitoring-plugin/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.13868174Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-blv5n/monitoring-plugin/monitoring-plugin/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.138780972Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-blv5n/monitoring-plugin/monitoring-plugin/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.138840753Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-blv5n/monitoring-plugin/monitoring-plugin/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.138885825Z namespaces/openshift-monitoring/pods/node-exporter-6ps9j/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.138929455Z namespaces/openshift-monitoring/pods/node-exporter-6ps9j/node-exporter-6ps9j.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.139030937Z namespaces/openshift-monitoring/pods/node-exporter-6ps9j/init-textfile/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.139041908Z namespaces/openshift-monitoring/pods/node-exporter-6ps9j/init-textfile/init-textfile/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.139047818Z namespaces/openshift-monitoring/pods/node-exporter-6ps9j/init-textfile/init-textfile/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.139080108Z namespaces/openshift-monitoring/pods/node-exporter-6ps9j/init-textfile/init-textfile/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.13914383Z namespaces/openshift-monitoring/pods/node-exporter-6ps9j/init-textfile/init-textfile/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.139222821Z namespaces/openshift-monitoring/pods/node-exporter-6ps9j/init-textfile/init-textfile/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.139258152Z namespaces/openshift-monitoring/pods/node-exporter-6ps9j/kube-rbac-proxy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.139268412Z namespaces/openshift-monitoring/pods/node-exporter-6ps9j/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.139278173Z namespaces/openshift-monitoring/pods/node-exporter-6ps9j/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.139318673Z namespaces/openshift-monitoring/pods/node-exporter-6ps9j/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.139406515Z namespaces/openshift-monitoring/pods/node-exporter-6ps9j/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.139480646Z namespaces/openshift-monitoring/pods/node-exporter-6ps9j/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.139520537Z namespaces/openshift-monitoring/pods/node-exporter-6ps9j/node-exporter/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.139528067Z namespaces/openshift-monitoring/pods/node-exporter-6ps9j/node-exporter/node-exporter/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.139539848Z namespaces/openshift-monitoring/pods/node-exporter-6ps9j/node-exporter/node-exporter/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.139593289Z namespaces/openshift-monitoring/pods/node-exporter-6ps9j/node-exporter/node-exporter/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.139719571Z namespaces/openshift-monitoring/pods/node-exporter-6ps9j/node-exporter/node-exporter/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.139778052Z namespaces/openshift-monitoring/pods/node-exporter-6ps9j/node-exporter/node-exporter/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.139845064Z namespaces/openshift-monitoring/pods/node-exporter-gxdc6/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.139877174Z namespaces/openshift-monitoring/pods/node-exporter-gxdc6/node-exporter-gxdc6.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.139984427Z namespaces/openshift-monitoring/pods/node-exporter-gxdc6/init-textfile/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.139995227Z namespaces/openshift-monitoring/pods/node-exporter-gxdc6/init-textfile/init-textfile/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.140001247Z namespaces/openshift-monitoring/pods/node-exporter-gxdc6/init-textfile/init-textfile/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.140015757Z namespaces/openshift-monitoring/pods/node-exporter-gxdc6/init-textfile/init-textfile/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.140095569Z namespaces/openshift-monitoring/pods/node-exporter-gxdc6/init-textfile/init-textfile/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.14016207Z namespaces/openshift-monitoring/pods/node-exporter-gxdc6/init-textfile/init-textfile/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.140196011Z namespaces/openshift-monitoring/pods/node-exporter-gxdc6/kube-rbac-proxy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.140204751Z namespaces/openshift-monitoring/pods/node-exporter-gxdc6/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.140209521Z namespaces/openshift-monitoring/pods/node-exporter-gxdc6/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.140236512Z namespaces/openshift-monitoring/pods/node-exporter-gxdc6/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.140331314Z namespaces/openshift-monitoring/pods/node-exporter-gxdc6/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.140395465Z namespaces/openshift-monitoring/pods/node-exporter-gxdc6/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.140440476Z namespaces/openshift-monitoring/pods/node-exporter-gxdc6/node-exporter/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.140449696Z namespaces/openshift-monitoring/pods/node-exporter-gxdc6/node-exporter/node-exporter/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.140454986Z namespaces/openshift-monitoring/pods/node-exporter-gxdc6/node-exporter/node-exporter/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.140464826Z namespaces/openshift-monitoring/pods/node-exporter-gxdc6/node-exporter/node-exporter/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.140602279Z namespaces/openshift-monitoring/pods/node-exporter-gxdc6/node-exporter/node-exporter/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.140666921Z namespaces/openshift-monitoring/pods/node-exporter-gxdc6/node-exporter/node-exporter/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.140703691Z namespaces/openshift-monitoring/pods/node-exporter-mttd2/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.140742222Z namespaces/openshift-monitoring/pods/node-exporter-mttd2/node-exporter-mttd2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.140846654Z namespaces/openshift-monitoring/pods/node-exporter-mttd2/init-textfile/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.140859514Z namespaces/openshift-monitoring/pods/node-exporter-mttd2/init-textfile/init-textfile/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.140864674Z namespaces/openshift-monitoring/pods/node-exporter-mttd2/init-textfile/init-textfile/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.140876855Z namespaces/openshift-monitoring/pods/node-exporter-mttd2/init-textfile/init-textfile/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.140963776Z namespaces/openshift-monitoring/pods/node-exporter-mttd2/init-textfile/init-textfile/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.141025878Z namespaces/openshift-monitoring/pods/node-exporter-mttd2/init-textfile/init-textfile/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.141067859Z namespaces/openshift-monitoring/pods/node-exporter-mttd2/kube-rbac-proxy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.141075469Z namespaces/openshift-monitoring/pods/node-exporter-mttd2/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.141079429Z namespaces/openshift-monitoring/pods/node-exporter-mttd2/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.1411244Z namespaces/openshift-monitoring/pods/node-exporter-mttd2/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.141202401Z namespaces/openshift-monitoring/pods/node-exporter-mttd2/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.141268103Z namespaces/openshift-monitoring/pods/node-exporter-mttd2/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.141297613Z namespaces/openshift-monitoring/pods/node-exporter-mttd2/node-exporter/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.141303183Z namespaces/openshift-monitoring/pods/node-exporter-mttd2/node-exporter/node-exporter/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.141309134Z namespaces/openshift-monitoring/pods/node-exporter-mttd2/node-exporter/node-exporter/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.141354684Z namespaces/openshift-monitoring/pods/node-exporter-mttd2/node-exporter/node-exporter/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.141461067Z namespaces/openshift-monitoring/pods/node-exporter-mttd2/node-exporter/node-exporter/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.141518918Z namespaces/openshift-monitoring/pods/node-exporter-mttd2/node-exporter/node-exporter/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.141567999Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-g2lz6/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.14160777Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-g2lz6/openshift-state-metrics-b8f49f5c6-g2lz6.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.141765493Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-g2lz6/kube-rbac-proxy-main/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.141777333Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-g2lz6/kube-rbac-proxy-main/kube-rbac-proxy-main/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.141781653Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-g2lz6/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.141875235Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-g2lz6/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.141960547Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-g2lz6/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.142020588Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-g2lz6/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.142063319Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-g2lz6/kube-rbac-proxy-self/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.142072839Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-g2lz6/kube-rbac-proxy-self/kube-rbac-proxy-self/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.142083619Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-g2lz6/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.14212572Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-g2lz6/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.142213812Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-g2lz6/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.142286183Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-g2lz6/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.142300924Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-g2lz6/openshift-state-metrics/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.142308714Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-g2lz6/openshift-state-metrics/openshift-state-metrics/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.142347495Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-g2lz6/openshift-state-metrics/openshift-state-metrics/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.142391276Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-g2lz6/openshift-state-metrics/openshift-state-metrics/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.142496988Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-g2lz6/openshift-state-metrics/openshift-state-metrics/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.14260279Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-g2lz6/openshift-state-metrics/openshift-state-metrics/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.142664171Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.142721282Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus-k8s-0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.142919266Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.142928367Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.142933356Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.142972377Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.143075759Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.143139231Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.143179641Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.143187032Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.143194662Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.143241063Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.143493978Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.143559139Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.14361634Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.143628601Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.143632861Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.143663611Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.143747943Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.143820274Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.143858615Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.143874556Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.143887866Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.143899396Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.143984638Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.144059339Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.14407199Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.1440807Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.14408468Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.144128521Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.144220553Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.144285454Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.144318805Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.144326485Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.144333165Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.144369316Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.14506929Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.145151041Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.145175282Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.145185762Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.145193742Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.145231073Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.145362006Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.145430297Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.145476748Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-ngzcw/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.145501289Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-ngzcw/prometheus-operator-84d69f794d-ngzcw.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.145659572Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-ngzcw/kube-rbac-proxy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.145675562Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-ngzcw/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.145681622Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-ngzcw/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.145707613Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-ngzcw/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.145796285Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-ngzcw/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.145862146Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-ngzcw/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.145896257Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-ngzcw/prometheus-operator/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.145903577Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-ngzcw/prometheus-operator/prometheus-operator/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.145907457Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-ngzcw/prometheus-operator/prometheus-operator/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.145951708Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-ngzcw/prometheus-operator/prometheus-operator/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.146223843Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-ngzcw/prometheus-operator/prometheus-operator/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.146292115Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-ngzcw/prometheus-operator/prometheus-operator/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.146343046Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-9pwjg/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.146371296Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-9pwjg/prometheus-operator-admission-webhook-7f8d78f54b-9pwjg.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.146504939Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-9pwjg/prometheus-operator-admission-webhook/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.146513969Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-9pwjg/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.146518179Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-9pwjg/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.14656895Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-9pwjg/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.146682212Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-9pwjg/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.146750714Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-9pwjg/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.146779474Z namespaces/openshift-monitoring/pods/telemeter-client-5fcc6cf86-n5c9g/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.146812695Z namespaces/openshift-monitoring/pods/telemeter-client-5fcc6cf86-n5c9g/telemeter-client-5fcc6cf86-n5c9g.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.146965928Z namespaces/openshift-monitoring/pods/telemeter-client-5fcc6cf86-n5c9g/kube-rbac-proxy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.146973038Z namespaces/openshift-monitoring/pods/telemeter-client-5fcc6cf86-n5c9g/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.146977149Z namespaces/openshift-monitoring/pods/telemeter-client-5fcc6cf86-n5c9g/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.147008399Z namespaces/openshift-monitoring/pods/telemeter-client-5fcc6cf86-n5c9g/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.147100001Z namespaces/openshift-monitoring/pods/telemeter-client-5fcc6cf86-n5c9g/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.147165553Z namespaces/openshift-monitoring/pods/telemeter-client-5fcc6cf86-n5c9g/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.147200963Z namespaces/openshift-monitoring/pods/telemeter-client-5fcc6cf86-n5c9g/reload/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.147208133Z namespaces/openshift-monitoring/pods/telemeter-client-5fcc6cf86-n5c9g/reload/reload/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.147212233Z namespaces/openshift-monitoring/pods/telemeter-client-5fcc6cf86-n5c9g/reload/reload/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.147256284Z namespaces/openshift-monitoring/pods/telemeter-client-5fcc6cf86-n5c9g/reload/reload/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.147344136Z namespaces/openshift-monitoring/pods/telemeter-client-5fcc6cf86-n5c9g/reload/reload/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.147413218Z namespaces/openshift-monitoring/pods/telemeter-client-5fcc6cf86-n5c9g/reload/reload/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.147454118Z namespaces/openshift-monitoring/pods/telemeter-client-5fcc6cf86-n5c9g/telemeter-client/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.147463399Z namespaces/openshift-monitoring/pods/telemeter-client-5fcc6cf86-n5c9g/telemeter-client/telemeter-client/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.147469459Z namespaces/openshift-monitoring/pods/telemeter-client-5fcc6cf86-n5c9g/telemeter-client/telemeter-client/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.1475101Z namespaces/openshift-monitoring/pods/telemeter-client-5fcc6cf86-n5c9g/telemeter-client/telemeter-client/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.147626462Z namespaces/openshift-monitoring/pods/telemeter-client-5fcc6cf86-n5c9g/telemeter-client/telemeter-client/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.147710173Z namespaces/openshift-monitoring/pods/telemeter-client-5fcc6cf86-n5c9g/telemeter-client/telemeter-client/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.147753574Z namespaces/openshift-monitoring/pods/thanos-querier-7d57b56d8-ws28t/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.147826546Z namespaces/openshift-monitoring/pods/thanos-querier-7d57b56d8-ws28t/thanos-querier-7d57b56d8-ws28t.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.147959818Z namespaces/openshift-monitoring/pods/thanos-querier-7d57b56d8-ws28t/kube-rbac-proxy-metrics/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.147967749Z namespaces/openshift-monitoring/pods/thanos-querier-7d57b56d8-ws28t/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.147971759Z namespaces/openshift-monitoring/pods/thanos-querier-7d57b56d8-ws28t/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.147992989Z namespaces/openshift-monitoring/pods/thanos-querier-7d57b56d8-ws28t/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.148090571Z namespaces/openshift-monitoring/pods/thanos-querier-7d57b56d8-ws28t/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.148155493Z namespaces/openshift-monitoring/pods/thanos-querier-7d57b56d8-ws28t/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.148199414Z namespaces/openshift-monitoring/pods/thanos-querier-7d57b56d8-ws28t/kube-rbac-proxy-rules/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.148206863Z namespaces/openshift-monitoring/pods/thanos-querier-7d57b56d8-ws28t/kube-rbac-proxy-rules/kube-rbac-proxy-rules/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.148217164Z namespaces/openshift-monitoring/pods/thanos-querier-7d57b56d8-ws28t/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.148249955Z namespaces/openshift-monitoring/pods/thanos-querier-7d57b56d8-ws28t/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.148337216Z namespaces/openshift-monitoring/pods/thanos-querier-7d57b56d8-ws28t/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.148399958Z namespaces/openshift-monitoring/pods/thanos-querier-7d57b56d8-ws28t/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.148421208Z namespaces/openshift-monitoring/pods/thanos-querier-7d57b56d8-ws28t/kube-rbac-proxy-web/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.148427488Z namespaces/openshift-monitoring/pods/thanos-querier-7d57b56d8-ws28t/kube-rbac-proxy-web/kube-rbac-proxy-web/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.148439428Z namespaces/openshift-monitoring/pods/thanos-querier-7d57b56d8-ws28t/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.148486029Z namespaces/openshift-monitoring/pods/thanos-querier-7d57b56d8-ws28t/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.148593882Z namespaces/openshift-monitoring/pods/thanos-querier-7d57b56d8-ws28t/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.148658663Z namespaces/openshift-monitoring/pods/thanos-querier-7d57b56d8-ws28t/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.148691133Z namespaces/openshift-monitoring/pods/thanos-querier-7d57b56d8-ws28t/kube-rbac-proxy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.148695873Z namespaces/openshift-monitoring/pods/thanos-querier-7d57b56d8-ws28t/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.148702074Z namespaces/openshift-monitoring/pods/thanos-querier-7d57b56d8-ws28t/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.148745885Z namespaces/openshift-monitoring/pods/thanos-querier-7d57b56d8-ws28t/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.148835916Z namespaces/openshift-monitoring/pods/thanos-querier-7d57b56d8-ws28t/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.148902848Z namespaces/openshift-monitoring/pods/thanos-querier-7d57b56d8-ws28t/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.148937478Z namespaces/openshift-monitoring/pods/thanos-querier-7d57b56d8-ws28t/prom-label-proxy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.148945418Z namespaces/openshift-monitoring/pods/thanos-querier-7d57b56d8-ws28t/prom-label-proxy/prom-label-proxy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.148951629Z namespaces/openshift-monitoring/pods/thanos-querier-7d57b56d8-ws28t/prom-label-proxy/prom-label-proxy/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.148961209Z namespaces/openshift-monitoring/pods/thanos-querier-7d57b56d8-ws28t/prom-label-proxy/prom-label-proxy/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.149061901Z namespaces/openshift-monitoring/pods/thanos-querier-7d57b56d8-ws28t/prom-label-proxy/prom-label-proxy/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.149126482Z namespaces/openshift-monitoring/pods/thanos-querier-7d57b56d8-ws28t/prom-label-proxy/prom-label-proxy/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.149153363Z namespaces/openshift-monitoring/pods/thanos-querier-7d57b56d8-ws28t/thanos-query/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.149165113Z namespaces/openshift-monitoring/pods/thanos-querier-7d57b56d8-ws28t/thanos-query/thanos-query/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.149171203Z namespaces/openshift-monitoring/pods/thanos-querier-7d57b56d8-ws28t/thanos-query/thanos-query/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.149226464Z namespaces/openshift-monitoring/pods/thanos-querier-7d57b56d8-ws28t/thanos-query/thanos-query/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.149328226Z namespaces/openshift-monitoring/pods/thanos-querier-7d57b56d8-ws28t/thanos-query/thanos-query/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.149388127Z namespaces/openshift-monitoring/pods/thanos-querier-7d57b56d8-ws28t/thanos-query/thanos-query/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.149429158Z namespaces/openshift-monitoring/policy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.149462759Z namespaces/openshift-monitoring/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.14951093Z namespaces/openshift-monitoring/route.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.149545391Z namespaces/openshift-monitoring/route.openshift.io/routes.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.149684513Z namespaces/openshift-multus/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.149718654Z namespaces/openshift-multus/openshift-multus.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.149858297Z namespaces/openshift-multus/apps.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.149891988Z namespaces/openshift-multus/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.149939169Z namespaces/openshift-multus/apps/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.14998499Z namespaces/openshift-multus/apps/daemonsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.150189474Z namespaces/openshift-multus/apps/deployments.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.150264845Z namespaces/openshift-multus/apps/replicasets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.150348977Z namespaces/openshift-multus/apps/statefulsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.150376638Z namespaces/openshift-multus/apps/daemonsets/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.150424619Z namespaces/openshift-multus/apps/daemonsets/multus-additional-cni-plugins.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.150819256Z namespaces/openshift-multus/apps/daemonsets/multus.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.15096297Z namespaces/openshift-multus/apps/daemonsets/network-metrics-daemon.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.151024041Z namespaces/openshift-multus/autoscaling/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.151060612Z namespaces/openshift-multus/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.151118423Z namespaces/openshift-multus/batch/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.151147573Z namespaces/openshift-multus/batch/cronjobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.151234485Z namespaces/openshift-multus/batch/jobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.151277616Z namespaces/openshift-multus/build.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.151325647Z namespaces/openshift-multus/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.151412159Z namespaces/openshift-multus/build.openshift.io/builds.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.1514671Z namespaces/openshift-multus/core/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.151511371Z namespaces/openshift-multus/core/configmaps.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.151673604Z namespaces/openshift-multus/core/endpoints.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.151757406Z namespaces/openshift-multus/core/events.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.152229935Z namespaces/openshift-multus/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.152352808Z namespaces/openshift-multus/core/pods.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.15297722Z namespaces/openshift-multus/core/replicationcontrollers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.153059592Z namespaces/openshift-multus/core/secrets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.153196255Z namespaces/openshift-multus/core/services.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.153262796Z namespaces/openshift-multus/core/configmaps/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.153300367Z namespaces/openshift-multus/core/configmaps/cni-copy-resources.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.153404329Z namespaces/openshift-multus/core/configmaps/default-cni-sysctl-allowlist.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.153484301Z namespaces/openshift-multus/core/configmaps/multus-daemon-config.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.153566322Z namespaces/openshift-multus/core/configmaps/whereabouts-flatfile-config.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.153680565Z namespaces/openshift-multus/core/serviceaccounts/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.153698235Z namespaces/openshift-multus/core/serviceaccounts/metrics-daemon-sa.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.153790807Z namespaces/openshift-multus/core/serviceaccounts/multus-ac.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.153875549Z namespaces/openshift-multus/core/serviceaccounts/multus-ancillary-tools.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.15396292Z namespaces/openshift-multus/core/serviceaccounts/multus.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.154025972Z namespaces/openshift-multus/core/services/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.154080153Z namespaces/openshift-multus/core/services/network-metrics-service.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.154138274Z namespaces/openshift-multus/discovery.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.154179545Z namespaces/openshift-multus/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.154254786Z namespaces/openshift-multus/image.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.154297317Z namespaces/openshift-multus/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.154347938Z namespaces/openshift-multus/k8s.ovn.org/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.15441658Z namespaces/openshift-multus/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.154506741Z namespaces/openshift-multus/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.154616604Z namespaces/openshift-multus/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.154671125Z namespaces/openshift-multus/monitoring.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.154721866Z namespaces/openshift-multus/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.154774007Z namespaces/openshift-multus/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.154812517Z namespaces/openshift-multus/monitoring.coreos.com/servicemonitors/monitor-network.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.154866539Z namespaces/openshift-multus/networking.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.154906539Z namespaces/openshift-multus/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.15495686Z namespaces/openshift-multus/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.154968081Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.155010041Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.155170895Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.155450821Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.155600473Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.155806548Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.155993481Z namespaces/openshift-multus/pods/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.156002522Z namespaces/openshift-multus/pods/multus-29wrb/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.156034342Z namespaces/openshift-multus/pods/multus-29wrb/multus-29wrb.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.156144115Z namespaces/openshift-multus/pods/multus-29wrb/kube-multus/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.156153735Z namespaces/openshift-multus/pods/multus-29wrb/kube-multus/kube-multus/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.156161235Z namespaces/openshift-multus/pods/multus-29wrb/kube-multus/kube-multus/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.156187945Z namespaces/openshift-multus/pods/multus-29wrb/kube-multus/kube-multus/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.156664905Z namespaces/openshift-multus/pods/multus-29wrb/kube-multus/kube-multus/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.156729516Z namespaces/openshift-multus/pods/multus-29wrb/kube-multus/kube-multus/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.156780028Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bmkbw/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.156823938Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bmkbw/multus-additional-cni-plugins-bmkbw.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.156966631Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bmkbw/bond-cni-plugin/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.156973392Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bmkbw/bond-cni-plugin/bond-cni-plugin/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.156977441Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bmkbw/bond-cni-plugin/bond-cni-plugin/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.157017772Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bmkbw/bond-cni-plugin/bond-cni-plugin/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.157105804Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bmkbw/bond-cni-plugin/bond-cni-plugin/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.157176376Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bmkbw/bond-cni-plugin/bond-cni-plugin/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.157197656Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bmkbw/cni-plugins/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.157203316Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bmkbw/cni-plugins/cni-plugins/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.157209266Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bmkbw/cni-plugins/cni-plugins/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.157252627Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bmkbw/cni-plugins/cni-plugins/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.157342759Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bmkbw/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.15740832Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bmkbw/cni-plugins/cni-plugins/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.157453251Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bmkbw/egress-router-binary-copy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.157462601Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bmkbw/egress-router-binary-copy/egress-router-binary-copy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.157481302Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bmkbw/egress-router-binary-copy/egress-router-binary-copy/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.157528843Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bmkbw/egress-router-binary-copy/egress-router-binary-copy/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.157634095Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bmkbw/egress-router-binary-copy/egress-router-binary-copy/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.157702216Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bmkbw/egress-router-binary-copy/egress-router-binary-copy/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.157739387Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bmkbw/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.157744777Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bmkbw/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.157748887Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bmkbw/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.157780078Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bmkbw/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.157847969Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bmkbw/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.157920691Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bmkbw/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.157954161Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bmkbw/routeoverride-cni/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.157961941Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bmkbw/routeoverride-cni/routeoverride-cni/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.157965972Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bmkbw/routeoverride-cni/routeoverride-cni/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.158005162Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bmkbw/routeoverride-cni/routeoverride-cni/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.158086334Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bmkbw/routeoverride-cni/routeoverride-cni/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.158150035Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bmkbw/routeoverride-cni/routeoverride-cni/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.158189416Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bmkbw/whereabouts-cni-bincopy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.158199696Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bmkbw/whereabouts-cni-bincopy/whereabouts-cni-bincopy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.158205696Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bmkbw/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.158217386Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bmkbw/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.158311508Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bmkbw/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.15838458Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bmkbw/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.158426101Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bmkbw/whereabouts-cni/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.158430281Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bmkbw/whereabouts-cni/whereabouts-cni/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.158434221Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bmkbw/whereabouts-cni/whereabouts-cni/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.158443251Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bmkbw/whereabouts-cni/whereabouts-cni/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.158544203Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bmkbw/whereabouts-cni/whereabouts-cni/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.158723557Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bmkbw/whereabouts-cni/whereabouts-cni/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.158742807Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fftp9/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.158798288Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fftp9/multus-additional-cni-plugins-fftp9.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.158929551Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fftp9/bond-cni-plugin/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.158937941Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fftp9/bond-cni-plugin/bond-cni-plugin/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.158941881Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fftp9/bond-cni-plugin/bond-cni-plugin/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.159002102Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fftp9/bond-cni-plugin/bond-cni-plugin/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.159086514Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fftp9/bond-cni-plugin/bond-cni-plugin/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.159154176Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fftp9/bond-cni-plugin/bond-cni-plugin/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.159192657Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fftp9/cni-plugins/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.159200617Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fftp9/cni-plugins/cni-plugins/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.159205557Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fftp9/cni-plugins/cni-plugins/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.159234517Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fftp9/cni-plugins/cni-plugins/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.159319569Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fftp9/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.15938256Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fftp9/cni-plugins/cni-plugins/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.159417231Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fftp9/egress-router-binary-copy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.159423581Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fftp9/egress-router-binary-copy/egress-router-binary-copy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.159427561Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fftp9/egress-router-binary-copy/egress-router-binary-copy/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.159469932Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fftp9/egress-router-binary-copy/egress-router-binary-copy/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.159588094Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fftp9/egress-router-binary-copy/egress-router-binary-copy/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.159666346Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fftp9/egress-router-binary-copy/egress-router-binary-copy/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.159693717Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fftp9/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.159704567Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fftp9/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.159708847Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fftp9/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.159743718Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fftp9/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.159814659Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fftp9/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.159883811Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fftp9/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.159909701Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fftp9/routeoverride-cni/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.159916451Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fftp9/routeoverride-cni/routeoverride-cni/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.159920481Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fftp9/routeoverride-cni/routeoverride-cni/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.159958062Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fftp9/routeoverride-cni/routeoverride-cni/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.160043204Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fftp9/routeoverride-cni/routeoverride-cni/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.160112035Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fftp9/routeoverride-cni/routeoverride-cni/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.160143686Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fftp9/whereabouts-cni-bincopy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.160150196Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fftp9/whereabouts-cni-bincopy/whereabouts-cni-bincopy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.160154056Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fftp9/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.160179706Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fftp9/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.160277228Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fftp9/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.16034394Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fftp9/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.160380361Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fftp9/whereabouts-cni/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.160387871Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fftp9/whereabouts-cni/whereabouts-cni/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.160396871Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fftp9/whereabouts-cni/whereabouts-cni/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.160434632Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fftp9/whereabouts-cni/whereabouts-cni/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.160522723Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fftp9/whereabouts-cni/whereabouts-cni/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.160598275Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fftp9/whereabouts-cni/whereabouts-cni/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.160674807Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gzgcl/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.160721348Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gzgcl/multus-additional-cni-plugins-gzgcl.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.16085897Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gzgcl/bond-cni-plugin/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.16086706Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gzgcl/bond-cni-plugin/bond-cni-plugin/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.160871131Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gzgcl/bond-cni-plugin/bond-cni-plugin/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.160904681Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gzgcl/bond-cni-plugin/bond-cni-plugin/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.160984893Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gzgcl/bond-cni-plugin/bond-cni-plugin/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.161050044Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gzgcl/bond-cni-plugin/bond-cni-plugin/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.161077175Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gzgcl/cni-plugins/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.161081765Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gzgcl/cni-plugins/cni-plugins/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.161085645Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gzgcl/cni-plugins/cni-plugins/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.161131056Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gzgcl/cni-plugins/cni-plugins/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.161220968Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gzgcl/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.161285299Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gzgcl/cni-plugins/cni-plugins/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.1613258Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gzgcl/egress-router-binary-copy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.16133442Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gzgcl/egress-router-binary-copy/egress-router-binary-copy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.16133855Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gzgcl/egress-router-binary-copy/egress-router-binary-copy/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.161362941Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gzgcl/egress-router-binary-copy/egress-router-binary-copy/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.161453142Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gzgcl/egress-router-binary-copy/egress-router-binary-copy/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.161518554Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gzgcl/egress-router-binary-copy/egress-router-binary-copy/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.161552004Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gzgcl/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.161557185Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gzgcl/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.161563595Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gzgcl/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.161612736Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gzgcl/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.161689377Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gzgcl/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.161884031Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gzgcl/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.161896791Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gzgcl/routeoverride-cni/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.161902832Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gzgcl/routeoverride-cni/routeoverride-cni/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.161908141Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gzgcl/routeoverride-cni/routeoverride-cni/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.161911942Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gzgcl/routeoverride-cni/routeoverride-cni/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.161936562Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gzgcl/routeoverride-cni/routeoverride-cni/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.162005094Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gzgcl/routeoverride-cni/routeoverride-cni/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.162036784Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gzgcl/whereabouts-cni-bincopy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.162043184Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gzgcl/whereabouts-cni-bincopy/whereabouts-cni-bincopy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.162047124Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gzgcl/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.162087165Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gzgcl/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.162172327Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gzgcl/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.162242948Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gzgcl/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.162278039Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gzgcl/whereabouts-cni/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.162283679Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gzgcl/whereabouts-cni/whereabouts-cni/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.162287729Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gzgcl/whereabouts-cni/whereabouts-cni/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.16231833Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gzgcl/whereabouts-cni/whereabouts-cni/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.162400142Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gzgcl/whereabouts-cni/whereabouts-cni/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.162469903Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gzgcl/whereabouts-cni/whereabouts-cni/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.162523464Z namespaces/openshift-multus/pods/multus-gtxhl/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.162593815Z namespaces/openshift-multus/pods/multus-gtxhl/multus-gtxhl.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.162693368Z namespaces/openshift-multus/pods/multus-gtxhl/kube-multus/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.162703488Z namespaces/openshift-multus/pods/multus-gtxhl/kube-multus/kube-multus/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.162709198Z namespaces/openshift-multus/pods/multus-gtxhl/kube-multus/kube-multus/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.162740489Z namespaces/openshift-multus/pods/multus-gtxhl/kube-multus/kube-multus/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.163601306Z namespaces/openshift-multus/pods/multus-gtxhl/kube-multus/kube-multus/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.163681177Z namespaces/openshift-multus/pods/multus-gtxhl/kube-multus/kube-multus/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.163719488Z namespaces/openshift-multus/pods/multus-p2rv8/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.163760679Z namespaces/openshift-multus/pods/multus-p2rv8/multus-p2rv8.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.163845281Z namespaces/openshift-multus/pods/multus-p2rv8/kube-multus/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.163853571Z namespaces/openshift-multus/pods/multus-p2rv8/kube-multus/kube-multus/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.163857861Z namespaces/openshift-multus/pods/multus-p2rv8/kube-multus/kube-multus/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.163891122Z namespaces/openshift-multus/pods/multus-p2rv8/kube-multus/kube-multus/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.164679068Z namespaces/openshift-multus/pods/multus-p2rv8/kube-multus/kube-multus/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.164746079Z namespaces/openshift-multus/pods/multus-p2rv8/kube-multus/kube-multus/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.16478978Z namespaces/openshift-multus/pods/network-metrics-daemon-299qd/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.164831551Z namespaces/openshift-multus/pods/network-metrics-daemon-299qd/network-metrics-daemon-299qd.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.164924253Z namespaces/openshift-multus/pods/network-metrics-daemon-299qd/kube-rbac-proxy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.164931803Z namespaces/openshift-multus/pods/network-metrics-daemon-299qd/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.164935863Z namespaces/openshift-multus/pods/network-metrics-daemon-299qd/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.164990694Z namespaces/openshift-multus/pods/network-metrics-daemon-299qd/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.165081706Z namespaces/openshift-multus/pods/network-metrics-daemon-299qd/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.165155587Z namespaces/openshift-multus/pods/network-metrics-daemon-299qd/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.165193228Z namespaces/openshift-multus/pods/network-metrics-daemon-299qd/network-metrics-daemon/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.165198628Z namespaces/openshift-multus/pods/network-metrics-daemon-299qd/network-metrics-daemon/network-metrics-daemon/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.165202428Z namespaces/openshift-multus/pods/network-metrics-daemon-299qd/network-metrics-daemon/network-metrics-daemon/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.165233979Z namespaces/openshift-multus/pods/network-metrics-daemon-299qd/network-metrics-daemon/network-metrics-daemon/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.165461514Z namespaces/openshift-multus/pods/network-metrics-daemon-299qd/network-metrics-daemon/network-metrics-daemon/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.165523785Z namespaces/openshift-multus/pods/network-metrics-daemon-299qd/network-metrics-daemon/network-metrics-daemon/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.165554415Z namespaces/openshift-multus/pods/network-metrics-daemon-8fltb/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.165623927Z namespaces/openshift-multus/pods/network-metrics-daemon-8fltb/network-metrics-daemon-8fltb.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.165718709Z namespaces/openshift-multus/pods/network-metrics-daemon-8fltb/kube-rbac-proxy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.165730699Z namespaces/openshift-multus/pods/network-metrics-daemon-8fltb/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.165736049Z namespaces/openshift-multus/pods/network-metrics-daemon-8fltb/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.16576718Z namespaces/openshift-multus/pods/network-metrics-daemon-8fltb/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.165886292Z namespaces/openshift-multus/pods/network-metrics-daemon-8fltb/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.165960914Z namespaces/openshift-multus/pods/network-metrics-daemon-8fltb/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.166002575Z namespaces/openshift-multus/pods/network-metrics-daemon-8fltb/network-metrics-daemon/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.166010495Z namespaces/openshift-multus/pods/network-metrics-daemon-8fltb/network-metrics-daemon/network-metrics-daemon/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.166018615Z namespaces/openshift-multus/pods/network-metrics-daemon-8fltb/network-metrics-daemon/network-metrics-daemon/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.166060326Z namespaces/openshift-multus/pods/network-metrics-daemon-8fltb/network-metrics-daemon/network-metrics-daemon/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.16626902Z namespaces/openshift-multus/pods/network-metrics-daemon-8fltb/network-metrics-daemon/network-metrics-daemon/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.166337681Z namespaces/openshift-multus/pods/network-metrics-daemon-8fltb/network-metrics-daemon/network-metrics-daemon/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.166376882Z namespaces/openshift-multus/pods/network-metrics-daemon-vtxgv/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.166403173Z namespaces/openshift-multus/pods/network-metrics-daemon-vtxgv/network-metrics-daemon-vtxgv.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.166489145Z namespaces/openshift-multus/pods/network-metrics-daemon-vtxgv/kube-rbac-proxy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.166496085Z namespaces/openshift-multus/pods/network-metrics-daemon-vtxgv/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.166500395Z namespaces/openshift-multus/pods/network-metrics-daemon-vtxgv/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.166587416Z namespaces/openshift-multus/pods/network-metrics-daemon-vtxgv/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.166735379Z namespaces/openshift-multus/pods/network-metrics-daemon-vtxgv/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.166802061Z namespaces/openshift-multus/pods/network-metrics-daemon-vtxgv/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.166820811Z namespaces/openshift-multus/pods/network-metrics-daemon-vtxgv/network-metrics-daemon/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.166828521Z namespaces/openshift-multus/pods/network-metrics-daemon-vtxgv/network-metrics-daemon/network-metrics-daemon/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.166841272Z namespaces/openshift-multus/pods/network-metrics-daemon-vtxgv/network-metrics-daemon/network-metrics-daemon/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.166889802Z namespaces/openshift-multus/pods/network-metrics-daemon-vtxgv/network-metrics-daemon/network-metrics-daemon/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.167079426Z namespaces/openshift-multus/pods/network-metrics-daemon-vtxgv/network-metrics-daemon/network-metrics-daemon/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.167144168Z namespaces/openshift-multus/pods/network-metrics-daemon-vtxgv/network-metrics-daemon/network-metrics-daemon/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.167183528Z namespaces/openshift-multus/policy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.167216849Z namespaces/openshift-multus/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.16726793Z namespaces/openshift-multus/rbac.authorization.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.16727969Z namespaces/openshift-multus/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.167306981Z namespaces/openshift-multus/rbac.authorization.k8s.io/rolebindings/multus-whereabouts.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.167394063Z namespaces/openshift-multus/rbac.authorization.k8s.io/rolebindings/prometheus-k8s.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.167441904Z namespaces/openshift-multus/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.167483175Z namespaces/openshift-multus/rbac.authorization.k8s.io/roles/prometheus-k8s.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.167564306Z namespaces/openshift-multus/rbac.authorization.k8s.io/roles/whereabouts-cni.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.167632508Z namespaces/openshift-multus/route.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.167669918Z namespaces/openshift-multus/route.openshift.io/routes.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.167717699Z namespaces/openshift-must-gather-wxwvv/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.16772613Z namespaces/openshift-must-gather-wxwvv/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.16773187Z namespaces/openshift-must-gather-wxwvv/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.16776924Z namespaces/openshift-must-gather-wxwvv/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.167926874Z namespaces/openshift-must-gather-wxwvv/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.168108637Z namespaces/openshift-must-gather-wxwvv/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.1682432Z namespaces/openshift-must-gather-wxwvv/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.168449974Z namespaces/openshift-must-gather-wxwvv/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.168658078Z namespaces/openshift-network-console/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.168700649Z namespaces/openshift-network-console/openshift-network-console.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.16875784Z namespaces/openshift-network-console/apps.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.168804121Z namespaces/openshift-network-console/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.168842202Z namespaces/openshift-network-console/apps/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.168885423Z namespaces/openshift-network-console/apps/daemonsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.168962835Z namespaces/openshift-network-console/apps/deployments.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.169073937Z namespaces/openshift-network-console/apps/replicasets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.169175649Z namespaces/openshift-network-console/apps/statefulsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.1692187Z namespaces/openshift-network-console/apps/deployments/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.169262451Z namespaces/openshift-network-console/apps/deployments/networking-console-plugin.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.169330702Z namespaces/openshift-network-console/autoscaling/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.169375563Z namespaces/openshift-network-console/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.169417594Z namespaces/openshift-network-console/batch/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.169439774Z namespaces/openshift-network-console/batch/cronjobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.169527286Z namespaces/openshift-network-console/batch/jobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.169587937Z namespaces/openshift-network-console/build.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.169629958Z namespaces/openshift-network-console/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.16971365Z namespaces/openshift-network-console/build.openshift.io/builds.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.169745331Z namespaces/openshift-network-console/core/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.169796461Z namespaces/openshift-network-console/core/configmaps.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.169906114Z namespaces/openshift-network-console/core/endpoints.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.169988445Z namespaces/openshift-network-console/core/events.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.170131498Z namespaces/openshift-network-console/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.17021184Z namespaces/openshift-network-console/core/pods.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.170346663Z namespaces/openshift-network-console/core/replicationcontrollers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.170475455Z namespaces/openshift-network-console/core/secrets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.170620368Z namespaces/openshift-network-console/core/services.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.17067401Z namespaces/openshift-network-console/core/configmaps/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.17072278Z namespaces/openshift-network-console/core/configmaps/networking-console-plugin.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.170772331Z namespaces/openshift-network-console/core/services/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.170812692Z namespaces/openshift-network-console/core/services/networking-console-plugin.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.170867933Z namespaces/openshift-network-console/discovery.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.170905294Z namespaces/openshift-network-console/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.170958735Z namespaces/openshift-network-console/image.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.170999776Z namespaces/openshift-network-console/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.171047877Z namespaces/openshift-network-console/k8s.ovn.org/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.171080928Z namespaces/openshift-network-console/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.171164339Z namespaces/openshift-network-console/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.171237641Z namespaces/openshift-network-console/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.171271422Z namespaces/openshift-network-console/monitoring.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.171380414Z namespaces/openshift-network-console/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.171438455Z namespaces/openshift-network-console/networking.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.171465115Z namespaces/openshift-network-console/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.171523336Z namespaces/openshift-network-console/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.171530417Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.171562837Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.171731651Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.172007486Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.172145269Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.172335823Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.172518747Z namespaces/openshift-network-console/pods/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.172527417Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-m4n6h/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.172562118Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-m4n6h/networking-console-plugin-74c9f884b7-m4n6h.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.17265629Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-m4n6h/networking-console-plugin/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.17266813Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-m4n6h/networking-console-plugin/networking-console-plugin/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.17267208Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-m4n6h/networking-console-plugin/networking-console-plugin/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.172701811Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-m4n6h/networking-console-plugin/networking-console-plugin/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.172794182Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-m4n6h/networking-console-plugin/networking-console-plugin/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.172860494Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-m4n6h/networking-console-plugin/networking-console-plugin/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.172902124Z namespaces/openshift-network-console/policy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.172915245Z namespaces/openshift-network-console/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.172974606Z namespaces/openshift-network-console/route.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.173021297Z namespaces/openshift-network-console/route.openshift.io/routes.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.173064588Z namespaces/openshift-network-diagnostics/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.173115619Z namespaces/openshift-network-diagnostics/openshift-network-diagnostics.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.17316335Z namespaces/openshift-network-diagnostics/apps.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.173213541Z namespaces/openshift-network-diagnostics/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.173263722Z namespaces/openshift-network-diagnostics/apps/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.173301763Z namespaces/openshift-network-diagnostics/apps/daemonsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.173404735Z namespaces/openshift-network-diagnostics/apps/deployments.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.173500277Z namespaces/openshift-network-diagnostics/apps/replicasets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.173592679Z namespaces/openshift-network-diagnostics/apps/statefulsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.17365641Z namespaces/openshift-network-diagnostics/apps/daemonsets/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.173702551Z namespaces/openshift-network-diagnostics/apps/daemonsets/network-check-target.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.173764982Z namespaces/openshift-network-diagnostics/apps/deployments/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.173801473Z namespaces/openshift-network-diagnostics/apps/deployments/network-check-source.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.173862104Z namespaces/openshift-network-diagnostics/autoscaling/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.173906685Z namespaces/openshift-network-diagnostics/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.173946986Z namespaces/openshift-network-diagnostics/batch/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.173990017Z namespaces/openshift-network-diagnostics/batch/cronjobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.174064578Z namespaces/openshift-network-diagnostics/batch/jobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.174099169Z namespaces/openshift-network-diagnostics/build.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.17415281Z namespaces/openshift-network-diagnostics/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.174243962Z namespaces/openshift-network-diagnostics/build.openshift.io/builds.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.174284123Z namespaces/openshift-network-diagnostics/core/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.174316533Z namespaces/openshift-network-diagnostics/core/configmaps.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.174421755Z namespaces/openshift-network-diagnostics/core/endpoints.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.174523028Z namespaces/openshift-network-diagnostics/core/events.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.174851964Z namespaces/openshift-network-diagnostics/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.174931946Z namespaces/openshift-network-diagnostics/core/pods.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.17513416Z namespaces/openshift-network-diagnostics/core/replicationcontrollers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.175216822Z namespaces/openshift-network-diagnostics/core/secrets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.175317664Z namespaces/openshift-network-diagnostics/core/services.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.175374025Z namespaces/openshift-network-diagnostics/core/serviceaccounts/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.175417486Z namespaces/openshift-network-diagnostics/core/serviceaccounts/network-diagnostics.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.175466907Z namespaces/openshift-network-diagnostics/core/services/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.175500447Z namespaces/openshift-network-diagnostics/core/services/network-check-source.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.175588639Z namespaces/openshift-network-diagnostics/core/services/network-check-target.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.17565792Z namespaces/openshift-network-diagnostics/discovery.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.175702052Z namespaces/openshift-network-diagnostics/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.175762013Z namespaces/openshift-network-diagnostics/image.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.175794413Z namespaces/openshift-network-diagnostics/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.175852264Z namespaces/openshift-network-diagnostics/k8s.ovn.org/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.175883805Z namespaces/openshift-network-diagnostics/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.175964837Z namespaces/openshift-network-diagnostics/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.176040808Z namespaces/openshift-network-diagnostics/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.176084799Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.17612519Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.176184691Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.176206202Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/servicemonitors/network-check-source.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.176261303Z namespaces/openshift-network-diagnostics/networking.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.176295984Z namespaces/openshift-network-diagnostics/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.176363365Z namespaces/openshift-network-diagnostics/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.176380735Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.176394626Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.176537588Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.176733952Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.176874755Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.177067159Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.177391936Z namespaces/openshift-network-diagnostics/pods/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.177400896Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-vf6dm/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.177425876Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-vf6dm/network-check-source-7d88cb55d8-vf6dm.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.177498928Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-vf6dm/check-endpoints/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.177507658Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-vf6dm/check-endpoints/check-endpoints/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.177512478Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-vf6dm/check-endpoints/check-endpoints/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.177553379Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-vf6dm/check-endpoints/check-endpoints/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.177661461Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-vf6dm/check-endpoints/check-endpoints/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.177739403Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-vf6dm/check-endpoints/check-endpoints/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.177763043Z namespaces/openshift-network-diagnostics/pods/network-check-target-4msch/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.177797434Z namespaces/openshift-network-diagnostics/pods/network-check-target-4msch/network-check-target-4msch.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.177881316Z namespaces/openshift-network-diagnostics/pods/network-check-target-4msch/network-check-target-container/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.177888846Z namespaces/openshift-network-diagnostics/pods/network-check-target-4msch/network-check-target-container/network-check-target-container/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.177893166Z namespaces/openshift-network-diagnostics/pods/network-check-target-4msch/network-check-target-container/network-check-target-container/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.177923297Z namespaces/openshift-network-diagnostics/pods/network-check-target-4msch/network-check-target-container/network-check-target-container/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.178009738Z namespaces/openshift-network-diagnostics/pods/network-check-target-4msch/network-check-target-container/network-check-target-container/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.17807734Z namespaces/openshift-network-diagnostics/pods/network-check-target-4msch/network-check-target-container/network-check-target-container/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.17810538Z namespaces/openshift-network-diagnostics/pods/network-check-target-cmrjd/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.178147331Z namespaces/openshift-network-diagnostics/pods/network-check-target-cmrjd/network-check-target-cmrjd.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.178219143Z namespaces/openshift-network-diagnostics/pods/network-check-target-cmrjd/network-check-target-container/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.178226213Z namespaces/openshift-network-diagnostics/pods/network-check-target-cmrjd/network-check-target-container/network-check-target-container/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.178230173Z namespaces/openshift-network-diagnostics/pods/network-check-target-cmrjd/network-check-target-container/network-check-target-container/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.178250703Z namespaces/openshift-network-diagnostics/pods/network-check-target-cmrjd/network-check-target-container/network-check-target-container/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.178354315Z namespaces/openshift-network-diagnostics/pods/network-check-target-cmrjd/network-check-target-container/network-check-target-container/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.178419956Z namespaces/openshift-network-diagnostics/pods/network-check-target-cmrjd/network-check-target-container/network-check-target-container/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.178448697Z namespaces/openshift-network-diagnostics/pods/network-check-target-ddrhj/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.178492178Z namespaces/openshift-network-diagnostics/pods/network-check-target-ddrhj/network-check-target-ddrhj.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.17859023Z namespaces/openshift-network-diagnostics/pods/network-check-target-ddrhj/network-check-target-container/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.17860289Z namespaces/openshift-network-diagnostics/pods/network-check-target-ddrhj/network-check-target-container/network-check-target-container/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.178608591Z namespaces/openshift-network-diagnostics/pods/network-check-target-ddrhj/network-check-target-container/network-check-target-container/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.178644061Z namespaces/openshift-network-diagnostics/pods/network-check-target-ddrhj/network-check-target-container/network-check-target-container/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.178732923Z namespaces/openshift-network-diagnostics/pods/network-check-target-ddrhj/network-check-target-container/network-check-target-container/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.178799074Z namespaces/openshift-network-diagnostics/pods/network-check-target-ddrhj/network-check-target-container/network-check-target-container/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.178834815Z namespaces/openshift-network-diagnostics/policy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.178860636Z namespaces/openshift-network-diagnostics/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.178915547Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.178923767Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.178936467Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/rolebindings/network-diagnostics.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.179031249Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/rolebindings/prometheus-k8s.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.17905998Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.179106161Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/roles/network-diagnostics.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.179190492Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/roles/prometheus-k8s.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.179237713Z namespaces/openshift-network-diagnostics/route.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.179275424Z namespaces/openshift-network-diagnostics/route.openshift.io/routes.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.179321555Z namespaces/openshift-network-node-identity/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.179370106Z namespaces/openshift-network-node-identity/openshift-network-node-identity.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.179424877Z namespaces/openshift-network-node-identity/apps.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.179465888Z namespaces/openshift-network-node-identity/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.179512699Z namespaces/openshift-network-node-identity/apps/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.17955183Z namespaces/openshift-network-node-identity/apps/daemonsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.179651341Z namespaces/openshift-network-node-identity/apps/deployments.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.179729233Z namespaces/openshift-network-node-identity/apps/replicasets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.179805255Z namespaces/openshift-network-node-identity/apps/statefulsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.179854176Z namespaces/openshift-network-node-identity/autoscaling/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.179877996Z namespaces/openshift-network-node-identity/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.179934117Z namespaces/openshift-network-node-identity/batch/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.179966798Z namespaces/openshift-network-node-identity/batch/cronjobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.18004848Z namespaces/openshift-network-node-identity/batch/jobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.18009159Z namespaces/openshift-network-node-identity/build.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.180117761Z namespaces/openshift-network-node-identity/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.180205803Z namespaces/openshift-network-node-identity/build.openshift.io/builds.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.180243013Z namespaces/openshift-network-node-identity/coordination.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.180249524Z namespaces/openshift-network-node-identity/coordination.k8s.io/leases/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.180294334Z namespaces/openshift-network-node-identity/coordination.k8s.io/leases/ovnkube-identity.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.180341475Z namespaces/openshift-network-node-identity/core/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.180372956Z namespaces/openshift-network-node-identity/core/configmaps.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.180483088Z namespaces/openshift-network-node-identity/core/endpoints.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.18055706Z namespaces/openshift-network-node-identity/core/events.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.180661132Z namespaces/openshift-network-node-identity/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.180740663Z namespaces/openshift-network-node-identity/core/pods.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.180813565Z namespaces/openshift-network-node-identity/core/replicationcontrollers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.180951818Z namespaces/openshift-network-node-identity/core/secrets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.18104975Z namespaces/openshift-network-node-identity/core/services.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.181099751Z namespaces/openshift-network-node-identity/core/serviceaccounts/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.181126582Z namespaces/openshift-network-node-identity/core/serviceaccounts/network-node-identity.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.181184483Z namespaces/openshift-network-node-identity/discovery.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.181217613Z namespaces/openshift-network-node-identity/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.181268424Z namespaces/openshift-network-node-identity/image.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.181316765Z namespaces/openshift-network-node-identity/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.181374216Z namespaces/openshift-network-node-identity/k8s.ovn.org/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.181421537Z namespaces/openshift-network-node-identity/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.181493249Z namespaces/openshift-network-node-identity/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.181586301Z namespaces/openshift-network-node-identity/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.181633882Z namespaces/openshift-network-node-identity/monitoring.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.181675683Z namespaces/openshift-network-node-identity/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.181720983Z namespaces/openshift-network-node-identity/networking.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.181763404Z namespaces/openshift-network-node-identity/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.181795905Z namespaces/openshift-network-node-identity/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.181804535Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.181850216Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.18204006Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.182243774Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.182389697Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.182596841Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.182861747Z namespaces/openshift-network-node-identity/policy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.182931328Z namespaces/openshift-network-node-identity/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.182996319Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.183003669Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.18303107Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/rolebindings/network-node-identity-leases.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.183101931Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.183138972Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/roles/network-node-identity-leases.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.183190333Z namespaces/openshift-network-node-identity/route.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.183214414Z namespaces/openshift-network-node-identity/route.openshift.io/routes.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.183271855Z namespaces/openshift-network-operator/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.183313356Z namespaces/openshift-network-operator/openshift-network-operator.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.183369337Z namespaces/openshift-network-operator/apps.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.183409918Z namespaces/openshift-network-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.183443418Z namespaces/openshift-network-operator/apps/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.183493009Z namespaces/openshift-network-operator/apps/daemonsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.183624762Z namespaces/openshift-network-operator/apps/deployments.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.183712984Z namespaces/openshift-network-operator/apps/replicasets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.183788416Z namespaces/openshift-network-operator/apps/statefulsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.183837697Z namespaces/openshift-network-operator/apps/daemonsets/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.183873787Z namespaces/openshift-network-operator/apps/daemonsets/iptables-alerter.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.183938668Z namespaces/openshift-network-operator/autoscaling/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.183983459Z namespaces/openshift-network-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.18401298Z namespaces/openshift-network-operator/batch/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.184047441Z namespaces/openshift-network-operator/batch/cronjobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.184134752Z namespaces/openshift-network-operator/batch/jobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.184174223Z namespaces/openshift-network-operator/build.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.184218854Z namespaces/openshift-network-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.184297516Z namespaces/openshift-network-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.184341847Z namespaces/openshift-network-operator/coordination.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.184351497Z namespaces/openshift-network-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.184366757Z namespaces/openshift-network-operator/coordination.k8s.io/leases/network-operator-lock.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.184431209Z namespaces/openshift-network-operator/core/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.184466729Z namespaces/openshift-network-operator/core/configmaps.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.184592932Z namespaces/openshift-network-operator/core/endpoints.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.184699594Z namespaces/openshift-network-operator/core/events.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.184861497Z namespaces/openshift-network-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.184938029Z namespaces/openshift-network-operator/core/pods.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.185114142Z namespaces/openshift-network-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.185322647Z namespaces/openshift-network-operator/core/secrets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.185447529Z namespaces/openshift-network-operator/core/services.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.18550432Z namespaces/openshift-network-operator/core/configmaps/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.185540881Z namespaces/openshift-network-operator/core/configmaps/applied-cluster.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.185644723Z namespaces/openshift-network-operator/core/configmaps/iptables-alerter-script.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.185708054Z namespaces/openshift-network-operator/core/serviceaccounts/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.185760976Z namespaces/openshift-network-operator/core/serviceaccounts/iptables-alerter.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.185781246Z namespaces/openshift-network-operator/discovery.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.185833537Z namespaces/openshift-network-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.185877118Z namespaces/openshift-network-operator/image.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.185925999Z namespaces/openshift-network-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.185960629Z namespaces/openshift-network-operator/k8s.ovn.org/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.18600653Z namespaces/openshift-network-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.186079872Z namespaces/openshift-network-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.186159824Z namespaces/openshift-network-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.186201044Z namespaces/openshift-network-operator/monitoring.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.186240075Z namespaces/openshift-network-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.186295766Z namespaces/openshift-network-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.186336677Z namespaces/openshift-network-operator/monitoring.coreos.com/prometheusrules/openshift-network-operator-ipsec-rules.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.186381808Z namespaces/openshift-network-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.186436339Z namespaces/openshift-network-operator/monitoring.coreos.com/servicemonitors/network-operator.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.18647907Z namespaces/openshift-network-operator/networking.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.186517131Z namespaces/openshift-network-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.186565422Z namespaces/openshift-network-operator/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.186590452Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.186648394Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.186791386Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.18697088Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.187105573Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.187308437Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.18748884Z namespaces/openshift-network-operator/pods/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.187496161Z namespaces/openshift-network-operator/pods/iptables-alerter-bzknt/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.187531581Z namespaces/openshift-network-operator/pods/iptables-alerter-bzknt/iptables-alerter-bzknt.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.187649034Z namespaces/openshift-network-operator/pods/iptables-alerter-bzknt/iptables-alerter/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.187660354Z namespaces/openshift-network-operator/pods/iptables-alerter-bzknt/iptables-alerter/iptables-alerter/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.187664544Z namespaces/openshift-network-operator/pods/iptables-alerter-bzknt/iptables-alerter/iptables-alerter/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.187701815Z namespaces/openshift-network-operator/pods/iptables-alerter-bzknt/iptables-alerter/iptables-alerter/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.187811777Z namespaces/openshift-network-operator/pods/iptables-alerter-bzknt/iptables-alerter/iptables-alerter/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.187879759Z namespaces/openshift-network-operator/pods/iptables-alerter-bzknt/iptables-alerter/iptables-alerter/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.187926269Z namespaces/openshift-network-operator/pods/iptables-alerter-vgktv/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.18794206Z namespaces/openshift-network-operator/pods/iptables-alerter-vgktv/iptables-alerter-vgktv.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.188026351Z namespaces/openshift-network-operator/pods/iptables-alerter-vgktv/iptables-alerter/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.188034232Z namespaces/openshift-network-operator/pods/iptables-alerter-vgktv/iptables-alerter/iptables-alerter/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.188038332Z namespaces/openshift-network-operator/pods/iptables-alerter-vgktv/iptables-alerter/iptables-alerter/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.188064692Z namespaces/openshift-network-operator/pods/iptables-alerter-vgktv/iptables-alerter/iptables-alerter/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.188153154Z namespaces/openshift-network-operator/pods/iptables-alerter-vgktv/iptables-alerter/iptables-alerter/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.188225206Z namespaces/openshift-network-operator/pods/iptables-alerter-vgktv/iptables-alerter/iptables-alerter/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.188248366Z namespaces/openshift-network-operator/pods/iptables-alerter-whcvk/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.188293827Z namespaces/openshift-network-operator/pods/iptables-alerter-whcvk/iptables-alerter-whcvk.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.188366878Z namespaces/openshift-network-operator/pods/iptables-alerter-whcvk/iptables-alerter/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.188377898Z namespaces/openshift-network-operator/pods/iptables-alerter-whcvk/iptables-alerter/iptables-alerter/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.188383439Z namespaces/openshift-network-operator/pods/iptables-alerter-whcvk/iptables-alerter/iptables-alerter/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.188404179Z namespaces/openshift-network-operator/pods/iptables-alerter-whcvk/iptables-alerter/iptables-alerter/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.188491741Z namespaces/openshift-network-operator/pods/iptables-alerter-whcvk/iptables-alerter/iptables-alerter/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.188558012Z namespaces/openshift-network-operator/pods/iptables-alerter-whcvk/iptables-alerter/iptables-alerter/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.188600213Z namespaces/openshift-network-operator/policy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.188648294Z namespaces/openshift-network-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.188711785Z namespaces/openshift-network-operator/route.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.188754946Z namespaces/openshift-network-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.188791657Z namespaces/openshift-node/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.188797717Z namespaces/openshift-node/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.188801627Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.188855338Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.188997141Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.189177195Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.189317948Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.189518322Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.189726596Z namespaces/openshift-operator-lifecycle-manager/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.189754826Z namespaces/openshift-operator-lifecycle-manager/openshift-operator-lifecycle-manager.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.189836518Z namespaces/openshift-operator-lifecycle-manager/apps.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.189863709Z namespaces/openshift-operator-lifecycle-manager/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.18991623Z namespaces/openshift-operator-lifecycle-manager/apps/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.1899436Z namespaces/openshift-operator-lifecycle-manager/apps/daemonsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.190031702Z namespaces/openshift-operator-lifecycle-manager/apps/deployments.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.190106694Z namespaces/openshift-operator-lifecycle-manager/apps/replicasets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.190184935Z namespaces/openshift-operator-lifecycle-manager/apps/statefulsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.190224616Z namespaces/openshift-operator-lifecycle-manager/autoscaling/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.190259977Z namespaces/openshift-operator-lifecycle-manager/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.190310348Z namespaces/openshift-operator-lifecycle-manager/batch/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.190343498Z namespaces/openshift-operator-lifecycle-manager/batch/cronjobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.1904252Z namespaces/openshift-operator-lifecycle-manager/batch/jobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.190467971Z namespaces/openshift-operator-lifecycle-manager/build.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.190516102Z namespaces/openshift-operator-lifecycle-manager/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.190684515Z namespaces/openshift-operator-lifecycle-manager/build.openshift.io/builds.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.190740797Z namespaces/openshift-operator-lifecycle-manager/core/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.190756827Z namespaces/openshift-operator-lifecycle-manager/core/configmaps.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.190948561Z namespaces/openshift-operator-lifecycle-manager/core/endpoints.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.191029532Z namespaces/openshift-operator-lifecycle-manager/core/events.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.191121154Z namespaces/openshift-operator-lifecycle-manager/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.191192436Z namespaces/openshift-operator-lifecycle-manager/core/pods.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.191269087Z namespaces/openshift-operator-lifecycle-manager/core/replicationcontrollers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.19138955Z namespaces/openshift-operator-lifecycle-manager/core/secrets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.191588044Z namespaces/openshift-operator-lifecycle-manager/core/services.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.191654395Z namespaces/openshift-operator-lifecycle-manager/discovery.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.191697546Z namespaces/openshift-operator-lifecycle-manager/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.191745687Z namespaces/openshift-operator-lifecycle-manager/image.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.191794258Z namespaces/openshift-operator-lifecycle-manager/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.191837309Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.19187636Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.191953761Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.192027763Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.192073344Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.192115334Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.192160545Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.192204926Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/prometheusrules/olm-alert-rules.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.192264437Z namespaces/openshift-operator-lifecycle-manager/networking.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.192308758Z namespaces/openshift-operator-lifecycle-manager/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.19237545Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.1923852Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.192419021Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.192587524Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.192770688Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.192918691Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.193119725Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.193312039Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.193339649Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/operatorgroups/olm-operators.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.19340506Z namespaces/openshift-operator-lifecycle-manager/policy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.193431931Z namespaces/openshift-operator-lifecycle-manager/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.193490152Z namespaces/openshift-operator-lifecycle-manager/route.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.193517353Z namespaces/openshift-operator-lifecycle-manager/route.openshift.io/routes.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.193564744Z namespaces/openshift-operators/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.193584754Z namespaces/openshift-operators/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.193592614Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.193627755Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.193782008Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.193965812Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.194099615Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.194328719Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.194553294Z namespaces/openshift-operators/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.194621485Z namespaces/openshift-operators/operators.coreos.com/operatorgroups/global-operators.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.194678306Z namespaces/openshift-ovn-kubernetes/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.194722757Z namespaces/openshift-ovn-kubernetes/openshift-ovn-kubernetes.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.194777508Z namespaces/openshift-ovn-kubernetes/apps.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.194809539Z namespaces/openshift-ovn-kubernetes/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.19486368Z namespaces/openshift-ovn-kubernetes/apps/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.194900901Z namespaces/openshift-ovn-kubernetes/apps/daemonsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.195072854Z namespaces/openshift-ovn-kubernetes/apps/deployments.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.195145956Z namespaces/openshift-ovn-kubernetes/apps/replicasets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.195224328Z namespaces/openshift-ovn-kubernetes/apps/statefulsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.195258548Z namespaces/openshift-ovn-kubernetes/apps/daemonsets/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.195298609Z namespaces/openshift-ovn-kubernetes/apps/daemonsets/ovnkube-node.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.195524244Z namespaces/openshift-ovn-kubernetes/autoscaling/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.195553034Z namespaces/openshift-ovn-kubernetes/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.195628786Z namespaces/openshift-ovn-kubernetes/batch/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.195675347Z namespaces/openshift-ovn-kubernetes/batch/cronjobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.195740178Z namespaces/openshift-ovn-kubernetes/batch/jobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.195780259Z namespaces/openshift-ovn-kubernetes/build.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.19583269Z namespaces/openshift-ovn-kubernetes/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.195905671Z namespaces/openshift-ovn-kubernetes/build.openshift.io/builds.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.195952432Z namespaces/openshift-ovn-kubernetes/coordination.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.195960883Z namespaces/openshift-ovn-kubernetes/coordination.k8s.io/leases/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.195975393Z namespaces/openshift-ovn-kubernetes/coordination.k8s.io/leases/ovn-kubernetes-master.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.196041034Z namespaces/openshift-ovn-kubernetes/core/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.196078735Z namespaces/openshift-ovn-kubernetes/core/configmaps.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.196271319Z namespaces/openshift-ovn-kubernetes/core/endpoints.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.196354451Z namespaces/openshift-ovn-kubernetes/core/events.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.196681297Z namespaces/openshift-ovn-kubernetes/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.1967994Z namespaces/openshift-ovn-kubernetes/core/pods.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.197287999Z namespaces/openshift-ovn-kubernetes/core/replicationcontrollers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.197419392Z namespaces/openshift-ovn-kubernetes/core/secrets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.197593196Z namespaces/openshift-ovn-kubernetes/core/services.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.197646526Z namespaces/openshift-ovn-kubernetes/core/configmaps/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.197697718Z namespaces/openshift-ovn-kubernetes/core/configmaps/ovnkube-config.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.197784369Z namespaces/openshift-ovn-kubernetes/core/configmaps/ovnkube-script-lib.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.197897362Z namespaces/openshift-ovn-kubernetes/core/serviceaccounts/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.197931742Z namespaces/openshift-ovn-kubernetes/core/serviceaccounts/ovn-kubernetes-control-plane.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.198016684Z namespaces/openshift-ovn-kubernetes/core/serviceaccounts/ovn-kubernetes-node.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.198049625Z namespaces/openshift-ovn-kubernetes/core/services/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.198086095Z namespaces/openshift-ovn-kubernetes/core/services/ovn-kubernetes-node.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.198151537Z namespaces/openshift-ovn-kubernetes/discovery.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.198195968Z namespaces/openshift-ovn-kubernetes/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.198256349Z namespaces/openshift-ovn-kubernetes/image.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.19829895Z namespaces/openshift-ovn-kubernetes/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.1983303Z namespaces/openshift-ovn-kubernetes/k8s.cni.cncf.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.198338071Z namespaces/openshift-ovn-kubernetes/k8s.cni.cncf.io/network-attachment-definitions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.198374031Z namespaces/openshift-ovn-kubernetes/k8s.cni.cncf.io/network-attachment-definitions/default.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.198424942Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.198457903Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.198545155Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.198731219Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.19877468Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.19881408Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.198873382Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.198891562Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/prometheusrules/networking-rules.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.198990964Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.199008824Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/servicemonitors/monitor-ovn-node.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.199066895Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.199076726Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/operatorpkis/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.199098706Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/operatorpkis/ovn.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.199187448Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/operatorpkis/signer.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.199220989Z namespaces/openshift-ovn-kubernetes/networking.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.19927348Z namespaces/openshift-ovn-kubernetes/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.19931524Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.199323331Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.199367821Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.199529535Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.199732549Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.199955664Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.200151877Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.200336251Z namespaces/openshift-ovn-kubernetes/pods/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.200343941Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4sg48/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.200405933Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4sg48/ovnkube-node-4sg48.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.200599066Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4sg48/kube-rbac-proxy-node/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.200614937Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4sg48/kube-rbac-proxy-node/kube-rbac-proxy-node/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.200621067Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4sg48/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.200649318Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4sg48/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.20076462Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4sg48/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.200829221Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4sg48/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.200856812Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4sg48/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.200861522Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4sg48/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.200865542Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4sg48/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.200913383Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4sg48/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.201028975Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4sg48/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.201099287Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4sg48/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.201114427Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4sg48/nbdb/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.201124517Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4sg48/nbdb/nbdb/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.201132277Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4sg48/nbdb/nbdb/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.201170198Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4sg48/nbdb/nbdb/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.20126453Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4sg48/nbdb/nbdb/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.201335792Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4sg48/nbdb/nbdb/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.201373892Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4sg48/northd/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.201384122Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4sg48/northd/northd/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.201390262Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4sg48/northd/northd/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.201397403Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4sg48/northd/northd/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.201513115Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4sg48/northd/northd/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.201594837Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4sg48/northd/northd/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.201616407Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4sg48/ovn-acl-logging/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.201622787Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4sg48/ovn-acl-logging/ovn-acl-logging/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.201634658Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4sg48/ovn-acl-logging/ovn-acl-logging/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.201676728Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4sg48/ovn-acl-logging/ovn-acl-logging/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.20176456Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4sg48/ovn-acl-logging/ovn-acl-logging/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.201833811Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4sg48/ovn-acl-logging/ovn-acl-logging/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.201859612Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4sg48/ovn-controller/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.201864022Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4sg48/ovn-controller/ovn-controller/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.201869082Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4sg48/ovn-controller/ovn-controller/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.201914693Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4sg48/ovn-controller/ovn-controller/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.202171869Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4sg48/ovn-controller/ovn-controller/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.20223434Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4sg48/ovn-controller/ovn-controller/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.202274691Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4sg48/ovnkube-controller/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.202284881Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4sg48/ovnkube-controller/ovnkube-controller/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.202321581Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4sg48/ovnkube-controller/ovnkube-controller/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.202380713Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4sg48/ovnkube-controller/ovnkube-controller/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.204053127Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4sg48/ovnkube-controller/ovnkube-controller/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.204119878Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4sg48/ovnkube-controller/ovnkube-controller/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.204159229Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4sg48/sbdb/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.204174929Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4sg48/sbdb/sbdb/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.204180929Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4sg48/sbdb/sbdb/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.20419943Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4sg48/sbdb/sbdb/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.204295921Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4sg48/sbdb/sbdb/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.204361083Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4sg48/sbdb/sbdb/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.204390674Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cjlcq/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.204458155Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cjlcq/ovnkube-node-cjlcq.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.204633178Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cjlcq/kube-rbac-proxy-node/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.204644449Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cjlcq/kube-rbac-proxy-node/kube-rbac-proxy-node/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.204648679Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cjlcq/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.204675829Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cjlcq/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.204792282Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cjlcq/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.204843943Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cjlcq/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.204884164Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cjlcq/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.204891974Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cjlcq/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.204895904Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cjlcq/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.204928754Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cjlcq/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.205027996Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cjlcq/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.205092608Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cjlcq/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.205139159Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cjlcq/nbdb/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.205149449Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cjlcq/nbdb/nbdb/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.205154979Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cjlcq/nbdb/nbdb/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.205163589Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cjlcq/nbdb/nbdb/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.205266521Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cjlcq/nbdb/nbdb/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.205338423Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cjlcq/nbdb/nbdb/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.205354653Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cjlcq/northd/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.205362653Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cjlcq/northd/northd/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.205373433Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cjlcq/northd/northd/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.205412464Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cjlcq/northd/northd/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.205513236Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cjlcq/northd/northd/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.205590078Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cjlcq/northd/northd/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.205626348Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cjlcq/ovn-acl-logging/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.205631108Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cjlcq/ovn-acl-logging/ovn-acl-logging/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.205635019Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cjlcq/ovn-acl-logging/ovn-acl-logging/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.205660479Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cjlcq/ovn-acl-logging/ovn-acl-logging/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.205759851Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cjlcq/ovn-acl-logging/ovn-acl-logging/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.205823132Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cjlcq/ovn-acl-logging/ovn-acl-logging/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.205858473Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cjlcq/ovn-controller/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.205865693Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cjlcq/ovn-controller/ovn-controller/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.205869733Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cjlcq/ovn-controller/ovn-controller/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.205895904Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cjlcq/ovn-controller/ovn-controller/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.206280122Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cjlcq/ovn-controller/ovn-controller/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.206344803Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cjlcq/ovn-controller/ovn-controller/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.206372194Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cjlcq/ovnkube-controller/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.206377674Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cjlcq/ovnkube-controller/ovnkube-controller/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.206383524Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cjlcq/ovnkube-controller/ovnkube-controller/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.206428755Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cjlcq/ovnkube-controller/ovnkube-controller/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.208449936Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cjlcq/ovnkube-controller/ovnkube-controller/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.208519477Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cjlcq/ovnkube-controller/ovnkube-controller/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.208534838Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cjlcq/sbdb/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.208541798Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cjlcq/sbdb/sbdb/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.208548078Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cjlcq/sbdb/sbdb/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.208604789Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cjlcq/sbdb/sbdb/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.208705801Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cjlcq/sbdb/sbdb/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.208772222Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cjlcq/sbdb/sbdb/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.208816163Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdqls/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.208868014Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdqls/ovnkube-node-hdqls.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.209023157Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdqls/kube-rbac-proxy-node/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.209031328Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdqls/kube-rbac-proxy-node/kube-rbac-proxy-node/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.209035378Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdqls/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.209060288Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdqls/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.2091558Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdqls/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.209235442Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdqls/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.209263782Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdqls/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.209271602Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdqls/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.209275942Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdqls/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.209312233Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdqls/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.209419885Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdqls/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.209487057Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdqls/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.209525218Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdqls/nbdb/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.209532258Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdqls/nbdb/nbdb/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.209536258Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdqls/nbdb/nbdb/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.209561148Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdqls/nbdb/nbdb/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.209705961Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdqls/nbdb/nbdb/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.209759672Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdqls/nbdb/nbdb/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.209778263Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdqls/northd/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.209784793Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdqls/northd/northd/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.209793283Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdqls/northd/northd/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.209834724Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdqls/northd/northd/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.209935836Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdqls/northd/northd/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.210001297Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdqls/northd/northd/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.210019468Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdqls/ovn-acl-logging/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.210030618Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdqls/ovn-acl-logging/ovn-acl-logging/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.210036428Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdqls/ovn-acl-logging/ovn-acl-logging/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.210076169Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdqls/ovn-acl-logging/ovn-acl-logging/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.21016182Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdqls/ovn-acl-logging/ovn-acl-logging/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.210232862Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdqls/ovn-acl-logging/ovn-acl-logging/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.210252662Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdqls/ovn-controller/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.210261742Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdqls/ovn-controller/ovn-controller/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.210271413Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdqls/ovn-controller/ovn-controller/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.210310974Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdqls/ovn-controller/ovn-controller/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.210715152Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdqls/ovn-controller/ovn-controller/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.210768963Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdqls/ovn-controller/ovn-controller/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.210818094Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdqls/ovnkube-controller/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.210827854Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdqls/ovnkube-controller/ovnkube-controller/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.210832974Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdqls/ovnkube-controller/ovnkube-controller/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.210840534Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdqls/ovnkube-controller/ovnkube-controller/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.212659871Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdqls/ovnkube-controller/ovnkube-controller/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.212725983Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdqls/ovnkube-controller/ovnkube-controller/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.212752033Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdqls/sbdb/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.212760023Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdqls/sbdb/sbdb/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.212764923Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdqls/sbdb/sbdb/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.212805244Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdqls/sbdb/sbdb/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.212904366Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdqls/sbdb/sbdb/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.212969928Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdqls/sbdb/sbdb/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.212993838Z namespaces/openshift-ovn-kubernetes/policy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.213045519Z namespaces/openshift-ovn-kubernetes/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.21306555Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.21307626Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.213125811Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.213214662Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/openshift-ovn-kubernetes-nodes-identity-limited.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.213340785Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/prometheus-k8s.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.213385736Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.213418217Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.213509179Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/openshift-ovn-kubernetes-node-limited.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.21359109Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/prometheus-k8s.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.213662262Z namespaces/openshift-ovn-kubernetes/route.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.213704232Z namespaces/openshift-ovn-kubernetes/route.openshift.io/routes.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.213748443Z namespaces/openshift-route-controller-manager/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.213755673Z namespaces/openshift-route-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.213759904Z namespaces/openshift-route-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.213782354Z namespaces/openshift-route-controller-manager/coordination.k8s.io/leases/openshift-route-controllers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.213842135Z namespaces/openshift-route-controller-manager/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.213853285Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.213867246Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.21407143Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.214277764Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.214431157Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.214668452Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.214890807Z namespaces/openshift-service-ca-operator/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.214934167Z namespaces/openshift-service-ca-operator/openshift-service-ca-operator.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.214997449Z namespaces/openshift-service-ca-operator/apps.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.215037079Z namespaces/openshift-service-ca-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.2150865Z namespaces/openshift-service-ca-operator/apps/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.215125001Z namespaces/openshift-service-ca-operator/apps/daemonsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.215203473Z namespaces/openshift-service-ca-operator/apps/deployments.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.215310475Z namespaces/openshift-service-ca-operator/apps/replicasets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.215404897Z namespaces/openshift-service-ca-operator/apps/statefulsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.215444938Z namespaces/openshift-service-ca-operator/autoscaling/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.215488719Z namespaces/openshift-service-ca-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.21553294Z namespaces/openshift-service-ca-operator/batch/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.21556642Z namespaces/openshift-service-ca-operator/batch/cronjobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.215730524Z namespaces/openshift-service-ca-operator/batch/jobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.215753064Z namespaces/openshift-service-ca-operator/build.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.215809865Z namespaces/openshift-service-ca-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.215889947Z namespaces/openshift-service-ca-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.215934378Z namespaces/openshift-service-ca-operator/coordination.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.215943828Z namespaces/openshift-service-ca-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.215980049Z namespaces/openshift-service-ca-operator/coordination.k8s.io/leases/service-ca-operator-lock.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.21602693Z namespaces/openshift-service-ca-operator/core/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.21607084Z namespaces/openshift-service-ca-operator/core/configmaps.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.216195603Z namespaces/openshift-service-ca-operator/core/endpoints.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.216276864Z namespaces/openshift-service-ca-operator/core/events.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.216471128Z namespaces/openshift-service-ca-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.21655644Z namespaces/openshift-service-ca-operator/core/pods.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.216701023Z namespaces/openshift-service-ca-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.216831506Z namespaces/openshift-service-ca-operator/core/secrets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.217002349Z namespaces/openshift-service-ca-operator/core/services.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.2170579Z namespaces/openshift-service-ca-operator/discovery.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.217086281Z namespaces/openshift-service-ca-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.217149482Z namespaces/openshift-service-ca-operator/image.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.217185863Z namespaces/openshift-service-ca-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.217229654Z namespaces/openshift-service-ca-operator/k8s.ovn.org/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.217257444Z namespaces/openshift-service-ca-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.217349146Z namespaces/openshift-service-ca-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.217424388Z namespaces/openshift-service-ca-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.217465239Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.217497889Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.217565281Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.217629232Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/servicemonitors/service-ca-operator.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.217676673Z namespaces/openshift-service-ca-operator/networking.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.217721134Z namespaces/openshift-service-ca-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.217773595Z namespaces/openshift-service-ca-operator/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.217781745Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.217803066Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.217953039Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.218258395Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.218395868Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.218603882Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.218812706Z namespaces/openshift-service-ca-operator/pods/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.218823606Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d6c75f84f-gn75p/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.218837247Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d6c75f84f-gn75p/service-ca-operator-5d6c75f84f-gn75p.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.218930058Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d6c75f84f-gn75p/service-ca-operator/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.218937238Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d6c75f84f-gn75p/service-ca-operator/service-ca-operator/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.218941289Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d6c75f84f-gn75p/service-ca-operator/service-ca-operator/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.218978169Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d6c75f84f-gn75p/service-ca-operator/service-ca-operator/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.219131382Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d6c75f84f-gn75p/service-ca-operator/service-ca-operator/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.219287616Z namespaces/openshift-service-ca-operator/policy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.219303746Z namespaces/openshift-service-ca-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.219359877Z namespaces/openshift-service-ca-operator/route.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.219389758Z namespaces/openshift-service-ca-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.219447879Z namespaces/openshift-service-ca/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.219480009Z namespaces/openshift-service-ca/openshift-service-ca.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.219538651Z namespaces/openshift-service-ca/apps.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.219583332Z namespaces/openshift-service-ca/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.219650163Z namespaces/openshift-service-ca/apps/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.219667653Z namespaces/openshift-service-ca/apps/daemonsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.219761665Z namespaces/openshift-service-ca/apps/deployments.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.219855567Z namespaces/openshift-service-ca/apps/replicasets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.219949159Z namespaces/openshift-service-ca/apps/statefulsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.21998258Z namespaces/openshift-service-ca/autoscaling/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.22001898Z namespaces/openshift-service-ca/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.220068692Z namespaces/openshift-service-ca/batch/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.220095722Z namespaces/openshift-service-ca/batch/cronjobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.220182874Z namespaces/openshift-service-ca/batch/jobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.220230755Z namespaces/openshift-service-ca/build.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.220267976Z namespaces/openshift-service-ca/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.220343717Z namespaces/openshift-service-ca/build.openshift.io/builds.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.220392208Z namespaces/openshift-service-ca/coordination.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.220399488Z namespaces/openshift-service-ca/coordination.k8s.io/leases/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.220435479Z namespaces/openshift-service-ca/coordination.k8s.io/leases/service-ca-controller-lock.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.22048163Z namespaces/openshift-service-ca/core/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.220522481Z namespaces/openshift-service-ca/core/configmaps.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.220640173Z namespaces/openshift-service-ca/core/endpoints.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.220713435Z namespaces/openshift-service-ca/core/events.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.220824397Z namespaces/openshift-service-ca/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.220900509Z namespaces/openshift-service-ca/core/pods.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.22100566Z namespaces/openshift-service-ca/core/replicationcontrollers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.221125833Z namespaces/openshift-service-ca/core/secrets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.221297177Z namespaces/openshift-service-ca/core/services.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.221362658Z namespaces/openshift-service-ca/discovery.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.221409309Z namespaces/openshift-service-ca/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.22145632Z namespaces/openshift-service-ca/image.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.22149915Z namespaces/openshift-service-ca/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.221542351Z namespaces/openshift-service-ca/k8s.ovn.org/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.221587182Z namespaces/openshift-service-ca/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.221694565Z namespaces/openshift-service-ca/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.221774256Z namespaces/openshift-service-ca/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.221820767Z namespaces/openshift-service-ca/monitoring.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.221850158Z namespaces/openshift-service-ca/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.221908549Z namespaces/openshift-service-ca/networking.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.22194783Z namespaces/openshift-service-ca/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.2219768Z namespaces/openshift-service-ca/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.22198343Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.222028311Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.222177564Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.222364378Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.222524251Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.222809877Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.223064332Z namespaces/openshift-service-ca/pods/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.223072233Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-8727m/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.223092453Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-8727m/service-ca-658d7b7c78-8727m.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.223179315Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-8727m/service-ca-controller/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.223186505Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-8727m/service-ca-controller/service-ca-controller/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.223190845Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-8727m/service-ca-controller/service-ca-controller/logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.223220425Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-8727m/service-ca-controller/service-ca-controller/logs/current.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.22347059Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-8727m/service-ca-controller/service-ca-controller/logs/previous.insecure.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.223536862Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-8727m/service-ca-controller/service-ca-controller/logs/previous.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.223586163Z namespaces/openshift-service-ca/policy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.223626374Z namespaces/openshift-service-ca/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.223680425Z namespaces/openshift-service-ca/route.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.223707505Z namespaces/openshift-service-ca/route.openshift.io/routes.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.223756926Z namespaces/openshift-user-workload-monitoring/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.223795557Z namespaces/openshift-user-workload-monitoring/openshift-user-workload-monitoring.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.223901059Z namespaces/openshift-user-workload-monitoring/apps.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.22393108Z namespaces/openshift-user-workload-monitoring/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.223983151Z namespaces/openshift-user-workload-monitoring/apps/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.224009341Z namespaces/openshift-user-workload-monitoring/apps/daemonsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.224098713Z namespaces/openshift-user-workload-monitoring/apps/deployments.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.224178445Z namespaces/openshift-user-workload-monitoring/apps/replicasets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.224258046Z namespaces/openshift-user-workload-monitoring/apps/statefulsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.224300867Z namespaces/openshift-user-workload-monitoring/autoscaling/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.224374689Z namespaces/openshift-user-workload-monitoring/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.22442237Z namespaces/openshift-user-workload-monitoring/batch/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.22444783Z namespaces/openshift-user-workload-monitoring/batch/cronjobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.224526122Z namespaces/openshift-user-workload-monitoring/batch/jobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.224594553Z namespaces/openshift-user-workload-monitoring/build.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.224620174Z namespaces/openshift-user-workload-monitoring/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.224710916Z namespaces/openshift-user-workload-monitoring/build.openshift.io/builds.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.224752476Z namespaces/openshift-user-workload-monitoring/core/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.224796148Z namespaces/openshift-user-workload-monitoring/core/configmaps.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.224893649Z namespaces/openshift-user-workload-monitoring/core/endpoints.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.224969841Z namespaces/openshift-user-workload-monitoring/core/events.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.225048333Z namespaces/openshift-user-workload-monitoring/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.225123704Z namespaces/openshift-user-workload-monitoring/core/pods.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.225202156Z namespaces/openshift-user-workload-monitoring/core/replicationcontrollers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.225321408Z namespaces/openshift-user-workload-monitoring/core/secrets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.225473271Z namespaces/openshift-user-workload-monitoring/core/services.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.225526552Z namespaces/openshift-user-workload-monitoring/discovery.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.225564723Z namespaces/openshift-user-workload-monitoring/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.225635534Z namespaces/openshift-user-workload-monitoring/image.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.225678375Z namespaces/openshift-user-workload-monitoring/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.225726576Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.225765477Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.225850249Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.22592544Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.225965441Z namespaces/openshift-user-workload-monitoring/monitoring.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.226014022Z namespaces/openshift-user-workload-monitoring/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.226051643Z namespaces/openshift-user-workload-monitoring/networking.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.226095204Z namespaces/openshift-user-workload-monitoring/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.226129244Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.226138975Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.226175445Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.226328568Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.226518883Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.226681516Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.22688557Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.227068634Z namespaces/openshift-user-workload-monitoring/policy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.227107204Z namespaces/openshift-user-workload-monitoring/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.227154025Z namespaces/openshift-user-workload-monitoring/route.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.227186766Z namespaces/openshift-user-workload-monitoring/route.openshift.io/routes.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.227238607Z namespaces/openshift/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.227270658Z namespaces/openshift/openshift.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.227331659Z namespaces/openshift/apps.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.22737828Z namespaces/openshift/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.227424951Z namespaces/openshift/apps/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.227454441Z namespaces/openshift/apps/daemonsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.227568594Z namespaces/openshift/apps/deployments.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.227671656Z namespaces/openshift/apps/replicasets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.227745527Z namespaces/openshift/apps/statefulsets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.227795398Z namespaces/openshift/autoscaling/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.227833689Z namespaces/openshift/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.227865Z namespaces/openshift/batch/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.227904811Z namespaces/openshift/batch/cronjobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.227988642Z namespaces/openshift/batch/jobs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.228020683Z namespaces/openshift/build.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.228070314Z namespaces/openshift/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.228149025Z namespaces/openshift/build.openshift.io/builds.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.228193926Z namespaces/openshift/core/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.228244777Z namespaces/openshift/core/configmaps.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.228328919Z namespaces/openshift/core/endpoints.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.228405261Z namespaces/openshift/core/events.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.228483322Z namespaces/openshift/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.228559634Z namespaces/openshift/core/pods.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.228657356Z namespaces/openshift/core/replicationcontrollers.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.228773908Z namespaces/openshift/core/secrets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.228923991Z namespaces/openshift/core/services.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.228964592Z namespaces/openshift/discovery.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.229002233Z namespaces/openshift/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.229038334Z namespaces/openshift/image.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.229072914Z namespaces/openshift/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.230450102Z namespaces/openshift/image.openshift.io/imagestreams/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.230495173Z namespaces/openshift/image.openshift.io/imagestreams/cli-artifacts.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.230612895Z namespaces/openshift/image.openshift.io/imagestreams/cli.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.230705167Z namespaces/openshift/image.openshift.io/imagestreams/dotnet-runtime.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.230807139Z namespaces/openshift/image.openshift.io/imagestreams/dotnet.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.230908572Z namespaces/openshift/image.openshift.io/imagestreams/driver-toolkit.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.230996323Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-eap-openshift-java11.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.231165767Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-eap-openshift.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.2313311Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-java-openshift.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.231453783Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-java11-openshift.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.231542774Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-karaf-openshift-jdk11.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.231656197Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-karaf-openshift.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.231780509Z namespaces/openshift/image.openshift.io/imagestreams/golang.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.231880381Z namespaces/openshift/image.openshift.io/imagestreams/httpd.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.231985623Z namespaces/openshift/image.openshift.io/imagestreams/installer-artifacts.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.232077555Z namespaces/openshift/image.openshift.io/imagestreams/installer.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.232163247Z namespaces/openshift/image.openshift.io/imagestreams/java-runtime.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.232265029Z namespaces/openshift/image.openshift.io/imagestreams/java.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.232380561Z namespaces/openshift/image.openshift.io/imagestreams/jboss-datagrid73-openshift.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.232492614Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp3-openjdk11-openshift.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.232592586Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp3-openjdk11-runtime-openshift.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.232696648Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp4-openjdk11-openshift.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.23279701Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp4-openjdk11-runtime-openshift.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.232888762Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk11-openshift.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.232988634Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk11-runtime-openshift.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.233086816Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk8-openshift.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.233179367Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk8-runtime-openshift.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.233270089Z namespaces/openshift/image.openshift.io/imagestreams/jboss-webserver57-openjdk11-tomcat9-openshift-ubi8.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.233376491Z namespaces/openshift/image.openshift.io/imagestreams/jboss-webserver57-openjdk8-tomcat9-openshift-ubi8.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.233492234Z namespaces/openshift/image.openshift.io/imagestreams/jenkins-agent-base.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.233584856Z namespaces/openshift/image.openshift.io/imagestreams/jenkins.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.233718209Z namespaces/openshift/image.openshift.io/imagestreams/mariadb.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.233823211Z namespaces/openshift/image.openshift.io/imagestreams/must-gather.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.233907492Z namespaces/openshift/image.openshift.io/imagestreams/mysql.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.234015204Z namespaces/openshift/image.openshift.io/imagestreams/network-tools.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.234101966Z namespaces/openshift/image.openshift.io/imagestreams/nginx.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.234221369Z namespaces/openshift/image.openshift.io/imagestreams/nodejs.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.234363502Z namespaces/openshift/image.openshift.io/imagestreams/oauth-proxy.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.234455844Z namespaces/openshift/image.openshift.io/imagestreams/openjdk-11-rhel7.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.234586676Z namespaces/openshift/image.openshift.io/imagestreams/perl.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.234705498Z namespaces/openshift/image.openshift.io/imagestreams/php.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.234815661Z namespaces/openshift/image.openshift.io/imagestreams/postgresql.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.234944393Z namespaces/openshift/image.openshift.io/imagestreams/postgresql13-for-sso75-openshift-rhel8.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.235034795Z namespaces/openshift/image.openshift.io/imagestreams/postgresql13-for-sso76-openshift-rhel8.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.235126647Z namespaces/openshift/image.openshift.io/imagestreams/python.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.235249349Z namespaces/openshift/image.openshift.io/imagestreams/redhat-openjdk18-openshift.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.235377292Z namespaces/openshift/image.openshift.io/imagestreams/redis.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.235482264Z namespaces/openshift/image.openshift.io/imagestreams/ruby.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.235594627Z namespaces/openshift/image.openshift.io/imagestreams/sso75-openshift-rhel8.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.235706299Z namespaces/openshift/image.openshift.io/imagestreams/sso76-openshift-rhel8.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.235792331Z namespaces/openshift/image.openshift.io/imagestreams/tests.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.235888693Z namespaces/openshift/image.openshift.io/imagestreams/tools.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.236001985Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-11-runtime.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.236137278Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-11.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.23625829Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-17-runtime.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.236367702Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-17.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.236488565Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-21-runtime.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.236594217Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-21.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.236696399Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-8-runtime.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.236811251Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-8.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.236905163Z namespaces/openshift/k8s.ovn.org/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.236936904Z namespaces/openshift/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.237023916Z namespaces/openshift/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.237100897Z namespaces/openshift/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.237138288Z namespaces/openshift/monitoring.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.237183479Z namespaces/openshift/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.23723274Z namespaces/openshift/networking.k8s.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.2372621Z namespaces/openshift/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.237299181Z namespaces/openshift/operators.coreos.com/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.237304421Z namespaces/openshift/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.237358432Z namespaces/openshift/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.237559716Z namespaces/openshift/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.237774781Z namespaces/openshift/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.237908074Z namespaces/openshift/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.238103227Z namespaces/openshift/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.238435104Z namespaces/openshift/policy/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.238468815Z namespaces/openshift/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.238517626Z namespaces/openshift/route.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.238564437Z namespaces/openshift/route.openshift.io/routes.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.23871383Z namespaces/openshift/template.openshift.io/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.23873523Z namespaces/openshift/template.openshift.io/templates/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.238749951Z namespaces/openshift/template.openshift.io/templates/cache-service.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.238850173Z namespaces/openshift/template.openshift.io/templates/cakephp-mysql-example.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.238968055Z namespaces/openshift/template.openshift.io/templates/cakephp-mysql-persistent.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.239078747Z namespaces/openshift/template.openshift.io/templates/dancer-mysql-example.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.239181959Z namespaces/openshift/template.openshift.io/templates/dancer-mysql-persistent.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.239340783Z namespaces/openshift/template.openshift.io/templates/datagrid-service.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.239444254Z namespaces/openshift/template.openshift.io/templates/django-psql-example.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.239558937Z namespaces/openshift/template.openshift.io/templates/django-psql-persistent.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.239681659Z namespaces/openshift/template.openshift.io/templates/eap-xp3-basic-s2i.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.239803102Z namespaces/openshift/template.openshift.io/templates/eap-xp4-basic-s2i.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.239912094Z namespaces/openshift/template.openshift.io/templates/eap74-basic-s2i.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.240023866Z namespaces/openshift/template.openshift.io/templates/eap74-https-s2i.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.240153329Z namespaces/openshift/template.openshift.io/templates/eap74-sso-s2i.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.240285422Z namespaces/openshift/template.openshift.io/templates/httpd-example.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.240394814Z namespaces/openshift/template.openshift.io/templates/jenkins-ephemeral-monitored.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.240528947Z namespaces/openshift/template.openshift.io/templates/jenkins-ephemeral.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.240654989Z namespaces/openshift/template.openshift.io/templates/jenkins-persistent-monitored.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.240764121Z namespaces/openshift/template.openshift.io/templates/jenkins-persistent.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.240868983Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk11-tomcat9-ubi8-basic-s2i.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.240994196Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk11-tomcat9-ubi8-https-s2i.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.241104128Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk8-tomcat9-ubi8-basic-s2i.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.24119853Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk8-tomcat9-ubi8-https-s2i.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.241301432Z namespaces/openshift/template.openshift.io/templates/mariadb-ephemeral.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.241398624Z namespaces/openshift/template.openshift.io/templates/mariadb-persistent.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.241501196Z namespaces/openshift/template.openshift.io/templates/mysql-ephemeral.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.241617609Z namespaces/openshift/template.openshift.io/templates/mysql-persistent.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.241719791Z namespaces/openshift/template.openshift.io/templates/nginx-example.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.241818733Z namespaces/openshift/template.openshift.io/templates/nodejs-postgresql-example.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.241931045Z namespaces/openshift/template.openshift.io/templates/nodejs-postgresql-persistent.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.242039767Z namespaces/openshift/template.openshift.io/templates/openjdk-web-basic-s2i.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.242141499Z namespaces/openshift/template.openshift.io/templates/postgresql-ephemeral.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.242243921Z namespaces/openshift/template.openshift.io/templates/postgresql-persistent.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.242341773Z namespaces/openshift/template.openshift.io/templates/rails-pgsql-persistent.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.242464786Z namespaces/openshift/template.openshift.io/templates/rails-postgresql-example.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.242594589Z namespaces/openshift/template.openshift.io/templates/react-web-app-example.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.242722341Z namespaces/openshift/template.openshift.io/templates/redis-ephemeral.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.242821573Z namespaces/openshift/template.openshift.io/templates/redis-persistent.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.242921135Z namespaces/openshift/template.openshift.io/templates/s2i-fuse712-spring-boot-2-camel-rest-3scale.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.243034567Z namespaces/openshift/template.openshift.io/templates/s2i-fuse712-spring-boot-2-camel-xml.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.24314436Z namespaces/openshift/template.openshift.io/templates/s2i-fuse712-spring-boot-2-camel.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.243245052Z namespaces/openshift/template.openshift.io/templates/sso75-https.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.243365764Z namespaces/openshift/template.openshift.io/templates/sso75-ocp4-x509-https.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.243473756Z namespaces/openshift/template.openshift.io/templates/sso75-ocp4-x509-postgresql-persistent.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.243607729Z namespaces/openshift/template.openshift.io/templates/sso75-postgresql-persistent.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.243738752Z namespaces/openshift/template.openshift.io/templates/sso75-postgresql.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.244032718Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-https.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.244181301Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-postgresql-persistent.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.244304263Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-postgresql.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.244427506Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-x509-https.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.244539478Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-x509-postgresql-persistent.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.24465733Z network_logs/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.244691001Z network_logs/cluster_scale [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.244789343Z network_logs/ippools.whereabouts.cni.cncf.io [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.244865295Z network_logs/multi-networkpolicy [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.244926026Z network_logs/net-attach-def [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.245022798Z network_logs/overlappingrangeipreservations.whereabouts.cni.cncf.io [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.24514877Z network_logs/ovn_kubernetes_top_pods [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.245251372Z network_logs/ovnk_database_store.tar.gz [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.24760329Z nodes/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.247658421Z nodes/debug [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.247757393Z nodes/ip-10-0-130-235.ec2.internal/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.247780444Z nodes/ip-10-0-130-235.ec2.internal/cpu_affinities.json [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.248217423Z nodes/ip-10-0-130-235.ec2.internal/dmesg [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.248489438Z nodes/ip-10-0-130-235.ec2.internal/ethtool_channels [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.24856341Z nodes/ip-10-0-130-235.ec2.internal/ethtool_features [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.248675362Z nodes/ip-10-0-130-235.ec2.internal/ip-10-0-130-235.ec2.internal_logs_kubelet.gz [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.249017249Z nodes/ip-10-0-130-235.ec2.internal/irq_affinities.json [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.249151902Z nodes/ip-10-0-130-235.ec2.internal/lscpu [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.249269434Z nodes/ip-10-0-130-235.ec2.internal/lspci [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.249364056Z nodes/ip-10-0-130-235.ec2.internal/podresources.json [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.249467548Z nodes/ip-10-0-130-235.ec2.internal/pods_info.json [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.249602931Z nodes/ip-10-0-130-235.ec2.internal/proc_cmdline [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.249714823Z nodes/ip-10-0-130-235.ec2.internal/sysinfo.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.251395037Z nodes/ip-10-0-130-235.ec2.internal/sysinfo.tgz [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.25153692Z nodes/ip-10-0-131-88.ec2.internal/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.25155408Z nodes/ip-10-0-131-88.ec2.internal/cpu_affinities.json [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.251910598Z nodes/ip-10-0-131-88.ec2.internal/dmesg [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.252183753Z nodes/ip-10-0-131-88.ec2.internal/ethtool_channels [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.252256435Z nodes/ip-10-0-131-88.ec2.internal/ethtool_features [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.252335146Z nodes/ip-10-0-131-88.ec2.internal/ip-10-0-131-88.ec2.internal_logs_kubelet.gz [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.252872257Z nodes/ip-10-0-131-88.ec2.internal/irq_affinities.json [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.25300932Z nodes/ip-10-0-131-88.ec2.internal/lscpu [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.253129182Z nodes/ip-10-0-131-88.ec2.internal/lspci [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.253230664Z nodes/ip-10-0-131-88.ec2.internal/podresources.json [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.253333076Z nodes/ip-10-0-131-88.ec2.internal/pods_info.json [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.253446309Z nodes/ip-10-0-131-88.ec2.internal/proc_cmdline [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.25352776Z nodes/ip-10-0-131-88.ec2.internal/sysinfo.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.255364928Z nodes/ip-10-0-131-88.ec2.internal/sysinfo.tgz [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.255605353Z nodes/ip-10-0-141-50.ec2.internal/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.255633443Z nodes/ip-10-0-141-50.ec2.internal/cpu_affinities.json [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.255912999Z nodes/ip-10-0-141-50.ec2.internal/dmesg [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.256158164Z nodes/ip-10-0-141-50.ec2.internal/ethtool_channels [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.256228535Z nodes/ip-10-0-141-50.ec2.internal/ethtool_features [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.256317907Z nodes/ip-10-0-141-50.ec2.internal/ip-10-0-141-50.ec2.internal_logs_kubelet.gz [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.256921719Z nodes/ip-10-0-141-50.ec2.internal/irq_affinities.json [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.257060542Z nodes/ip-10-0-141-50.ec2.internal/lscpu [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.257181465Z nodes/ip-10-0-141-50.ec2.internal/lspci [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.257271266Z nodes/ip-10-0-141-50.ec2.internal/podresources.json [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.257362078Z nodes/ip-10-0-141-50.ec2.internal/pods_info.json [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.257485091Z nodes/ip-10-0-141-50.ec2.internal/proc_cmdline [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.257565332Z nodes/ip-10-0-141-50.ec2.internal/sysinfo.log [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.259300597Z nodes/ip-10-0-141-50.ec2.internal/sysinfo.tgz [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.25944178Z pod_network_connectivity_check/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.259473231Z pod_network_connectivity_check/podnetworkconnectivitychecks.yaml [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.259597414Z static-pods/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.259611284Z static-pods/kube-apiserver/ [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.26881529Z [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.268832751Z sent 72,381 bytes received 8,705,290 bytes 5,851,780.67 bytes/sec [must-gather] [must-gather-hwxvc] OUT 2026-07-28T11:54:09.268838561Z total size is 108,633,018 speedup is 12.38 [must-gather] [must-gather ] OUT 2026-07-28T11:54:09.576724558Z namespace/openshift-must-gather-wxwvv deleted [must-gather] [must-gather] [must-gather] Reprinting Cluster State: [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: 9c77e1ff-462e-41e6-a513-a8204698bb2b [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.26" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [git-push-artifacts] WORK_DIR: /workspace/odh-ci-artifacts [git-push-artifacts] REPO_PATH: opendatahub-io/odh-build-metadata [git-push-artifacts] REPO_BRANCH: ci-artifacts [git-push-artifacts] SPARSE_FILE_PATH: test-artifacts/docs [git-push-artifacts] SOURCE_PATH: /workspace/artifacts-dir [git-push-artifacts] DEST_PATH: test-artifacts/kserve-group-test-kfrhh [git-push-artifacts] ALWAYS_PASS: false [git-push-artifacts] configuring gh token [git-push-artifacts] taking github token from Konflux bot [git-push-artifacts] Initialized empty Git repository in /workspace/odh-ci-artifacts/.git/ [git-push-artifacts] Using partial fetch with sparse checkout for: test-artifacts/docs [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] * [new branch] ci-artifacts -> origin/ci-artifacts [git-push-artifacts] Already on 'ci-artifacts' [git-push-artifacts] branch 'ci-artifacts' set up to track 'origin/ci-artifacts'. [git-push-artifacts] TASK_NAME=kserve-group-test-kfrhh-e2e-llm-inference-service [git-push-artifacts] PIPELINERUN_NAME=kserve-group-test-kfrhh [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] Already up to date. [git-push-artifacts] -rw-r--r--. 1 root 1001540000 16860767 Jul 28 11:55 /workspace/odh-ci-artifacts/test-artifacts/kserve-group-test-kfrhh/e2e-llm-inference-service.tar.gz [git-push-artifacts] [ci-artifacts 0b79ffd] Updating CI Artifacts in e2e-llm-inference-service [git-push-artifacts] 1 file changed, 0 insertions(+), 0 deletions(-) [git-push-artifacts] create mode 100644 test-artifacts/kserve-group-test-kfrhh/e2e-llm-inference-service.tar.gz [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] Already up to date. [git-push-artifacts] To https://github.com/opendatahub-io/odh-build-metadata.git [git-push-artifacts] 76f5d21..0b79ffd ci-artifacts -> ci-artifacts [fail-if-needed] Failing pipeline because deploy-and-e2e step failed container step-fail-if-needed has failed : [{"key":"StartedAt","value":"2026-07-28T11:56:00.679Z","type":3}]