task e2e-llm-inference-service has failed: "step-fail-if-needed" exited with code 1: Error [get-kubeconfig] Found kubeconfig secret: cluster-zm6s9-admin-kubeconfig [get-kubeconfig] Wrote kubeconfig to /credentials/cluster-zm6s9-kubeconfig [get-kubeconfig] Found admin password secret: cluster-zm6s9-admin-password [get-kubeconfig] Retrieved username [get-kubeconfig] Wrote password to /credentials/cluster-zm6s9-password [get-kubeconfig] API Server URL: https://a174c630cae34426492dbdbea3d47329-b34d48c7d6f8c17a.elb.us-east-1.amazonaws.com:6443 [get-kubeconfig] Console URL: https://console-openshift-console.apps.b989455d-6ee2-46a4-81de-c6abc6e0878d.prod.konfluxeaas.com [clone-repo] auto-sync/master-to-release-v0.17 [clone-repo] https://github.com/opendatahub-io/kserve [clone-repo] Cloning into '/workspace/source'... [clone-repo] Updating files: 46% (1509/3269) Updating files: 47% (1537/3269) Updating files: 48% (1570/3269) Updating files: 49% (1602/3269) Updating files: 50% (1635/3269) Updating files: 51% (1668/3269) Updating files: 52% (1700/3269) Updating files: 53% (1733/3269) Updating files: 54% (1766/3269) Updating files: 55% (1798/3269) Updating files: 56% (1831/3269) Updating files: 57% (1864/3269) Updating files: 58% (1897/3269) Updating files: 59% (1929/3269) Updating files: 60% (1962/3269) Updating files: 61% (1995/3269) Updating files: 62% (2027/3269) Updating files: 63% (2060/3269) Updating files: 64% (2093/3269) Updating files: 65% (2125/3269) Updating files: 66% (2158/3269) Updating files: 67% (2191/3269) Updating files: 68% (2223/3269) Updating files: 69% (2256/3269) Updating files: 70% (2289/3269) Updating files: 71% (2321/3269) Updating files: 72% (2354/3269) Updating files: 73% (2387/3269) Updating files: 74% (2420/3269) Updating files: 75% (2452/3269) Updating files: 76% (2485/3269) Updating files: 77% (2518/3269) Updating files: 78% (2550/3269) Updating files: 79% (2583/3269) Updating files: 80% (2616/3269) Updating files: 81% (2648/3269) Updating files: 82% (2681/3269) Updating files: 83% (2714/3269) Updating files: 84% (2746/3269) Updating files: 85% (2779/3269) Updating files: 86% (2812/3269) Updating files: 87% (2845/3269) Updating files: 88% (2877/3269) Updating files: 89% (2910/3269) Updating files: 90% (2943/3269) Updating files: 91% (2975/3269) Updating files: 92% (3008/3269) Updating files: 93% (3041/3269) Updating files: 94% (3073/3269) Updating files: 95% (3106/3269) Updating files: 96% (3139/3269) Updating files: 97% (3171/3269) Updating files: 98% (3204/3269) Updating files: 99% (3237/3269) Updating files: 100% (3269/3269) Updating files: 100% (3269/3269), done. [e2e-llm-inference-service] + bash [e2e-llm-inference-service] + STATUS_FILE=/test-status/deploy-and-e2e-status [e2e-llm-inference-service] + echo failed [e2e-llm-inference-service] + COMPONENT_NAME=kserve-agent-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-agent-ci '.[$component_name].image' [e2e-llm-inference-service] + export KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:da1a151aae0f419a312e9da5e6473fcbbaa6bf56d1033e65d294b56cc3bbac48 [e2e-llm-inference-service] + KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:da1a151aae0f419a312e9da5e6473fcbbaa6bf56d1033e65d294b56cc3bbac48 [e2e-llm-inference-service] + COMPONENT_NAME=kserve-controller-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-controller-ci '.[$component_name].image' [e2e-llm-inference-service] + export KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:6dbe90fc1e66fa697d36172c78c604df1b033c60023ad052bada7adb20987c06 [e2e-llm-inference-service] + KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:6dbe90fc1e66fa697d36172c78c604df1b033c60023ad052bada7adb20987c06 [e2e-llm-inference-service] + COMPONENT_NAME=kserve-router-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-router-ci '.[$component_name].image' [e2e-llm-inference-service] + export KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:41cb8bf9ff6c7df05622085595fdb4c1e1d3a571d74d4bdea37a3b72ad46799b [e2e-llm-inference-service] + KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:41cb8bf9ff6c7df05622085595fdb4c1e1d3a571d74d4bdea37a3b72ad46799b [e2e-llm-inference-service] + COMPONENT_NAME=kserve-storage-initializer-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-storage-initializer-ci '.[$component_name].image' [e2e-llm-inference-service] + export STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:13da3ee375cce7c9d4590e64ebb24ecf2e97ed4b2b64f9a4f2e3074ed4aafac1 [e2e-llm-inference-service] + STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:13da3ee375cce7c9d4590e64ebb24ecf2e97ed4b2b64f9a4f2e3074ed4aafac1 [e2e-llm-inference-service] + COMPONENT_NAME=odh-kserve-llmisvc-controller-ci [e2e-llm-inference-service] ++ jq -r --arg component_name odh-kserve-llmisvc-controller-ci '.[$component_name].image' [e2e-llm-inference-service] + export LLMISVC_CONTROLLER_IMAGE=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:c6eafbd6173b198967c1d458a4b4bb056f2d32a2d9c083284e09b22ad9a1149a [e2e-llm-inference-service] + LLMISVC_CONTROLLER_IMAGE=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:c6eafbd6173b198967c1d458a4b4bb056f2d32a2d9c083284e09b22ad9a1149a [e2e-llm-inference-service] + ./test/scripts/openshift-ci/run-e2e-tests.sh 'llminferenceservice and cluster_cpu and not autoscaling and not tracing' 2 llm-d [e2e-llm-inference-service] INFERENCE_POOL_GROUP=inference.networking.k8s.io (detected from OCP 4.21.26) [e2e-llm-inference-service] Installing on cluster [e2e-llm-inference-service] Using namespace: kserve for KServe components [e2e-llm-inference-service] SKLEARN_IMAGE=quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1828 [e2e-llm-inference-service] OPT_125M_MODEL_URI=s3://example-models/facebook/opt-125m [e2e-llm-inference-service] ERROR_404_ISVC_IMAGE=quay.io/opendatahub/error-404-isvc:odh-pr-1828 [e2e-llm-inference-service] SUCCESS_200_ISVC_IMAGE=quay.io/opendatahub/success-200-isvc:odh-pr-1828 [e2e-llm-inference-service] [INFO] Installing Kustomize v5.8.1 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed Kustomize v5.8.1 to /workspace/source/bin/kustomize [e2e-llm-inference-service] v5.8.1 [e2e-llm-inference-service] make: Entering directory '/workspace/source' [e2e-llm-inference-service] [INFO] Installing yq v4.52.1 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed yq v4.52.1 to /workspace/source/bin/yq [e2e-llm-inference-service] yq (https://github.com/mikefarah/yq/) version v4.52.1 [e2e-llm-inference-service] make: Leaving directory '/workspace/source' [e2e-llm-inference-service] Installing KServe Python SDK ... [e2e-llm-inference-service] [INFO] Installing uv 0.7.8 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed uv 0.7.8 to /workspace/source/bin/uv [e2e-llm-inference-service] warning: Failed to read project metadata (No `pyproject.toml` found in current directory or any parent directory). Running `uv self version` for compatibility. This fallback will be removed in the future; pass `--preview` to force an error. [e2e-llm-inference-service] uv 0.7.8 [e2e-llm-inference-service] Creating virtual environment... [e2e-llm-inference-service] warning: virtualenv's `--clear` has no effect (uv always clears the virtual environment) [e2e-llm-inference-service] Using CPython 3.9.25 interpreter at: /usr/bin/python3 [e2e-llm-inference-service] Creating virtual environment at: .venv [e2e-llm-inference-service] /workspace/source [e2e-llm-inference-service] Using CPython 3.11.13 interpreter at: /usr/bin/python3.11 [e2e-llm-inference-service] Creating virtual environment at: .venv [e2e-llm-inference-service] Resolved 274 packages in 1ms [e2e-llm-inference-service] Building kserve @ file:///workspace/source/python/kserve [e2e-llm-inference-service] Downloading portforward (3.9MiB) [e2e-llm-inference-service] Downloading pandas (12.5MiB) [e2e-llm-inference-service] Downloading pydantic-core (2.0MiB) [e2e-llm-inference-service] Downloading cryptography (4.5MiB) [e2e-llm-inference-service] Downloading kubernetes (1.9MiB) [e2e-llm-inference-service] Downloading uvloop (3.8MiB) [e2e-llm-inference-service] Downloading pyarrow (40.1MiB) [e2e-llm-inference-service] Downloading aiohttp (1.7MiB) [e2e-llm-inference-service] Downloading numpy (15.7MiB) [e2e-llm-inference-service] Downloading grpcio (6.4MiB) [e2e-llm-inference-service] Downloading setuptools (1.2MiB) [e2e-llm-inference-service] Downloading grpcio-tools (2.5MiB) [e2e-llm-inference-service] Downloading botocore (12.9MiB) [e2e-llm-inference-service] Downloading black (1.6MiB) [e2e-llm-inference-service] Downloading mypy (17.2MiB) [e2e-llm-inference-service] Building timeout-sampler==1.0.3 [e2e-llm-inference-service] Building python-simple-logger==2.0.19 [e2e-llm-inference-service] Downloading aiohttp [e2e-llm-inference-service] Downloading black [e2e-llm-inference-service] Downloading pydantic-core [e2e-llm-inference-service] Downloading grpcio-tools [e2e-llm-inference-service] Downloading setuptools [e2e-llm-inference-service] Built python-simple-logger==2.0.19 [e2e-llm-inference-service] Downloading portforward [e2e-llm-inference-service] Downloading uvloop [e2e-llm-inference-service] Downloading cryptography [e2e-llm-inference-service] Downloading grpcio [e2e-llm-inference-service] Downloading kubernetes [e2e-llm-inference-service] Built timeout-sampler==1.0.3 [e2e-llm-inference-service] Downloading numpy [e2e-llm-inference-service] Built kserve @ file:///workspace/source/python/kserve [e2e-llm-inference-service] Downloading pandas [e2e-llm-inference-service] Downloading botocore [e2e-llm-inference-service] Downloading pyarrow [e2e-llm-inference-service] Downloading mypy [e2e-llm-inference-service] Prepared 101 packages in 1.83s [e2e-llm-inference-service] warning: Failed to hardlink files; falling back to full copy. This may lead to degraded performance. [e2e-llm-inference-service] If the cache and target directories are on different filesystems, hardlinking may not be supported. [e2e-llm-inference-service] If this is intentional, set `export UV_LINK_MODE=copy` or use `--link-mode=copy` to suppress this warning. [e2e-llm-inference-service] Installed 101 packages in 508ms [e2e-llm-inference-service] + aiohappyeyeballs==2.6.1 [e2e-llm-inference-service] + aiohttp==3.14.1 [e2e-llm-inference-service] + aiosignal==1.4.0 [e2e-llm-inference-service] + annotated-doc==0.0.4 [e2e-llm-inference-service] + annotated-types==0.7.0 [e2e-llm-inference-service] + anyio==4.9.0 [e2e-llm-inference-service] + attrs==25.3.0 [e2e-llm-inference-service] + avro==1.12.0 [e2e-llm-inference-service] + black==24.3.0 [e2e-llm-inference-service] + boto3==1.37.35 [e2e-llm-inference-service] + botocore==1.37.35 [e2e-llm-inference-service] + cachetools==5.5.2 [e2e-llm-inference-service] + certifi==2025.1.31 [e2e-llm-inference-service] + cffi==2.0.0 [e2e-llm-inference-service] + charset-normalizer==3.4.1 [e2e-llm-inference-service] + click==8.4.2 [e2e-llm-inference-service] + cloudevents==1.11.0 [e2e-llm-inference-service] + colorama==0.4.6 [e2e-llm-inference-service] + colorlog==6.10.1 [e2e-llm-inference-service] + coverage==7.8.0 [e2e-llm-inference-service] + cryptography==49.0.0 [e2e-llm-inference-service] + deprecation==2.1.0 [e2e-llm-inference-service] + durationpy==0.9 [e2e-llm-inference-service] + execnet==2.1.1 [e2e-llm-inference-service] + fastapi==0.136.3 [e2e-llm-inference-service] + frozenlist==1.5.0 [e2e-llm-inference-service] + google-auth==2.39.0 [e2e-llm-inference-service] + grpc-interceptor==0.15.4 [e2e-llm-inference-service] + grpcio==1.78.1 [e2e-llm-inference-service] + grpcio-testing==1.78.1 [e2e-llm-inference-service] + grpcio-tools==1.78.1 [e2e-llm-inference-service] + h11==0.16.0 [e2e-llm-inference-service] + httpcore==1.0.9 [e2e-llm-inference-service] + httptools==0.6.4 [e2e-llm-inference-service] + httpx==0.27.2 [e2e-llm-inference-service] + httpx-retries==0.4.5 [e2e-llm-inference-service] + idna==3.10 [e2e-llm-inference-service] + iniconfig==2.1.0 [e2e-llm-inference-service] + jinja2==3.1.6 [e2e-llm-inference-service] + jmespath==1.0.1 [e2e-llm-inference-service] + kserve==0.20.0rc0 (from file:///workspace/source/python/kserve) [e2e-llm-inference-service] + kubernetes==32.0.1 [e2e-llm-inference-service] + markupsafe==3.0.2 [e2e-llm-inference-service] + multidict==6.4.3 [e2e-llm-inference-service] + mypy==0.991 [e2e-llm-inference-service] + mypy-extensions==1.0.0 [e2e-llm-inference-service] + numpy==2.2.4 [e2e-llm-inference-service] + oauthlib==3.2.2 [e2e-llm-inference-service] + orjson==3.11.9 [e2e-llm-inference-service] + packaging==24.2 [e2e-llm-inference-service] + pandas==2.2.3 [e2e-llm-inference-service] + pathspec==0.12.1 [e2e-llm-inference-service] + platformdirs==4.3.7 [e2e-llm-inference-service] + pluggy==1.5.0 [e2e-llm-inference-service] + portforward==0.7.1 [e2e-llm-inference-service] + prometheus-client==0.21.1 [e2e-llm-inference-service] + propcache==0.3.1 [e2e-llm-inference-service] + protobuf==6.33.5 [e2e-llm-inference-service] + psutil==5.9.8 [e2e-llm-inference-service] + pyarrow==19.0.1 [e2e-llm-inference-service] + pyasn1==0.6.3 [e2e-llm-inference-service] + pyasn1-modules==0.4.2 [e2e-llm-inference-service] + pycparser==2.22 [e2e-llm-inference-service] + pydantic==2.12.4 [e2e-llm-inference-service] + pydantic-core==2.41.5 [e2e-llm-inference-service] + pyjwt==2.13.0 [e2e-llm-inference-service] + pytest==7.4.4 [e2e-llm-inference-service] + pytest-asyncio==0.23.8 [e2e-llm-inference-service] + pytest-cov==5.0.0 [e2e-llm-inference-service] + pytest-httpx==0.30.0 [e2e-llm-inference-service] + pytest-json-report==1.5.0 [e2e-llm-inference-service] + pytest-metadata==3.1.1 [e2e-llm-inference-service] + pytest-xdist==3.6.1 [e2e-llm-inference-service] + python-dateutil==2.9.0.post0 [e2e-llm-inference-service] + python-dotenv==1.1.0 [e2e-llm-inference-service] + python-multipart==0.0.32 [e2e-llm-inference-service] + python-simple-logger==2.0.19 [e2e-llm-inference-service] + pytz==2025.2 [e2e-llm-inference-service] + pyyaml==6.0.2 [e2e-llm-inference-service] + requests==2.32.3 [e2e-llm-inference-service] + requests-oauthlib==2.0.0 [e2e-llm-inference-service] + rsa==4.9.1 [e2e-llm-inference-service] + s3transfer==0.11.4 [e2e-llm-inference-service] + setuptools==78.1.0 [e2e-llm-inference-service] + six==1.17.0 [e2e-llm-inference-service] + sniffio==1.3.1 [e2e-llm-inference-service] + starlette==1.2.1 [e2e-llm-inference-service] + tabulate==0.9.0 [e2e-llm-inference-service] + timeout-sampler==1.0.3 [e2e-llm-inference-service] + timing-asgi==0.3.1 [e2e-llm-inference-service] + tomlkit==0.13.2 [e2e-llm-inference-service] + typing-extensions==4.15.0 [e2e-llm-inference-service] + typing-inspection==0.4.2 [e2e-llm-inference-service] + tzdata==2025.2 [e2e-llm-inference-service] + urllib3==2.7.0 [e2e-llm-inference-service] + uvicorn==0.34.1 [e2e-llm-inference-service] + uvloop==0.21.0 [e2e-llm-inference-service] + watchfiles==1.0.5 [e2e-llm-inference-service] + websocket-client==1.8.0 [e2e-llm-inference-service] + websockets==15.0.1 [e2e-llm-inference-service] + yarl==1.20.0 [e2e-llm-inference-service] Audited 1 package in 47ms [e2e-llm-inference-service] /workspace/source [e2e-llm-inference-service] [INFO] Installing Kustomize v5.8.1 for linux/amd64... [e2e-llm-inference-service] [INFO] Kustomize v5.8.1 is already installed in /workspace/source/bin (>= v5.8.1) [e2e-llm-inference-service] make: Entering directory '/workspace/source' [e2e-llm-inference-service] make: Leaving directory '/workspace/source' [e2e-llm-inference-service] Now using project "kserve" on server "https://a174c630cae34426492dbdbea3d47329-b34d48c7d6f8c17a.elb.us-east-1.amazonaws.com:6443". [e2e-llm-inference-service] [e2e-llm-inference-service] You can add applications to this project with the 'new-app' command. For example, try: [e2e-llm-inference-service] [e2e-llm-inference-service] oc new-app rails-postgresql-example [e2e-llm-inference-service] [e2e-llm-inference-service] to build a new example application in Ruby. Or use kubectl to deploy a simple Kubernetes application: [e2e-llm-inference-service] [e2e-llm-inference-service] kubectl create deployment hello-node --image=registry.k8s.io/e2e-test-images/agnhost:2.43 -- /agnhost serve-hostname [e2e-llm-inference-service] [e2e-llm-inference-service] [INFO] Installing Kustomize v5.8.1 for linux/amd64... [e2e-llm-inference-service] [INFO] Kustomize v5.8.1 is already installed in /workspace/source/bin (>= v5.8.1) [e2e-llm-inference-service] make: Entering directory '/workspace/source' [e2e-llm-inference-service] make: Leaving directory '/workspace/source' [e2e-llm-inference-service] Creating namespace openshift-keda... [e2e-llm-inference-service] namespace/openshift-keda created [e2e-llm-inference-service] Namespace openshift-keda created/ensured. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Creating OperatorGroup openshift-keda... [e2e-llm-inference-service] operatorgroup.operators.coreos.com/openshift-keda created [e2e-llm-inference-service] OperatorGroup openshift-keda created/ensured. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Creating Subscription for openshift-custom-metrics-autoscaler-operator... [e2e-llm-inference-service] subscription.operators.coreos.com/openshift-custom-metrics-autoscaler-operator created [e2e-llm-inference-service] Subscription openshift-custom-metrics-autoscaler-operator created/ensured. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Waiting for openshift-custom-metrics-autoscaler-operator CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (0/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (5/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (10/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (15/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (20/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-2 found, but not yet Succeeded (Phase: Installing). Waiting... (25/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-2 found, but not yet Succeeded (Phase: Installing). Waiting... (30/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-2 found, but not yet Succeeded (Phase: Installing). Waiting... (35/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-2 found, but not yet Succeeded (Phase: Installing). Waiting... (40/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-2 found, but not yet Succeeded (Phase: Installing). Waiting... (45/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-2 found, but not yet Succeeded (Phase: Installing). Waiting... (50/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-2 found, but not yet Succeeded (Phase: Installing). Waiting... (55/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-2 is ready (Phase: Succeeded). [e2e-llm-inference-service] --- [e2e-llm-inference-service] Applying KedaController custom resource... [e2e-llm-inference-service] Warning: resource kedacontrollers/keda is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by oc apply. oc apply should only be used on resources created declaratively by either oc create --save-config or oc apply. The missing annotation will be patched automatically. [e2e-llm-inference-service] kedacontroller.keda.sh/keda configured [e2e-llm-inference-service] KedaController custom resource applied. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Allowing time for KEDA components to be provisioned by the operator ... [e2e-llm-inference-service] Waiting for KEDA Operator pod (selector: "app=keda-operator") to be ready in namespace openshift-keda... [e2e-llm-inference-service] Waiting for pod -l "app=keda-operator" in namespace "openshift-keda" to be created... [e2e-llm-inference-service] Pod -l "app=keda-operator" in namespace "openshift-keda" found. [e2e-llm-inference-service] Current pods for -l "app=keda-operator" in namespace "openshift-keda": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] keda-operator-77f648d694-vtqr8 1/1 Running 0 42s [e2e-llm-inference-service] Waiting up to 120s for pod(s) -l "app=keda-operator" in namespace "openshift-keda" to become ready... [e2e-llm-inference-service] pod/keda-operator-77f648d694-vtqr8 condition met [e2e-llm-inference-service] Pod(s) -l "app=keda-operator" in namespace "openshift-keda" are ready. [e2e-llm-inference-service] KEDA Operator pod is ready. [e2e-llm-inference-service] Waiting for KEDA Metrics API Server pod (selector: "app=keda-metrics-apiserver") to be ready in namespace openshift-keda... [e2e-llm-inference-service] Waiting for pod -l "app=keda-metrics-apiserver" in namespace "openshift-keda" to be created... [e2e-llm-inference-service] Pod -l "app=keda-metrics-apiserver" in namespace "openshift-keda" found. [e2e-llm-inference-service] Current pods for -l "app=keda-metrics-apiserver" in namespace "openshift-keda": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] keda-metrics-apiserver-55557df8c-hkqpw 1/1 Running 0 48s [e2e-llm-inference-service] Waiting up to 120s for pod(s) -l "app=keda-metrics-apiserver" in namespace "openshift-keda" to become ready... [e2e-llm-inference-service] pod/keda-metrics-apiserver-55557df8c-hkqpw condition met [e2e-llm-inference-service] Pod(s) -l "app=keda-metrics-apiserver" in namespace "openshift-keda" are ready. [e2e-llm-inference-service] KEDA Metrics API Server pod is ready. [e2e-llm-inference-service] Waiting for KEDA Webhook pod (selector: "app=keda-admission-webhooks") to be ready in namespace openshift-keda... [e2e-llm-inference-service] Waiting for pod -l "app=keda-admission-webhooks" in namespace "openshift-keda" to be created... [e2e-llm-inference-service] Pod -l "app=keda-admission-webhooks" in namespace "openshift-keda" found. [e2e-llm-inference-service] Current pods for -l "app=keda-admission-webhooks" in namespace "openshift-keda": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] keda-admission-56f9798bb4-t5v4l 1/1 Running 0 52s [e2e-llm-inference-service] Waiting up to 120s for pod(s) -l "app=keda-admission-webhooks" in namespace "openshift-keda" to become ready... [e2e-llm-inference-service] pod/keda-admission-56f9798bb4-t5v4l condition met [e2e-llm-inference-service] Pod(s) -l "app=keda-admission-webhooks" in namespace "openshift-keda" are ready. [e2e-llm-inference-service] KEDA Webhook pod is ready. [e2e-llm-inference-service] --- [e2e-llm-inference-service] ✅ KEDA deployment script finished successfully. [e2e-llm-inference-service] KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:6dbe90fc1e66fa697d36172c78c604df1b033c60023ad052bada7adb20987c06 [e2e-llm-inference-service] LLMISVC_CONTROLLER_IMAGE=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:c6eafbd6173b198967c1d458a4b4bb056f2d32a2d9c083284e09b22ad9a1149a [e2e-llm-inference-service] KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:da1a151aae0f419a312e9da5e6473fcbbaa6bf56d1033e65d294b56cc3bbac48 [e2e-llm-inference-service] KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:41cb8bf9ff6c7df05622085595fdb4c1e1d3a571d74d4bdea37a3b72ad46799b [e2e-llm-inference-service] STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:13da3ee375cce7c9d4590e64ebb24ecf2e97ed4b2b64f9a4f2e3074ed4aafac1 [e2e-llm-inference-service] Installing KServe via kustomize... [e2e-llm-inference-service] === Final params.env [e2e-llm-inference-service] kserve-controller=quay.io/opendatahub/kserve-controller@sha256:6dbe90fc1e66fa697d36172c78c604df1b033c60023ad052bada7adb20987c06 [e2e-llm-inference-service] llmisvc-controller=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:c6eafbd6173b198967c1d458a4b4bb056f2d32a2d9c083284e09b22ad9a1149a [e2e-llm-inference-service] kserve-agent=quay.io/opendatahub/kserve-agent@sha256:da1a151aae0f419a312e9da5e6473fcbbaa6bf56d1033e65d294b56cc3bbac48 [e2e-llm-inference-service] kserve-router=quay.io/opendatahub/kserve-router@sha256:41cb8bf9ff6c7df05622085595fdb4c1e1d3a571d74d4bdea37a3b72ad46799b [e2e-llm-inference-service] kserve-storage-initializer=quay.io/opendatahub/kserve-storage-initializer@sha256:13da3ee375cce7c9d4590e64ebb24ecf2e97ed4b2b64f9a4f2e3074ed4aafac1 [e2e-llm-inference-service] kserve-llm-d=registry.redhat.io/rhaiis/vllm-cuda-rhel9@sha256:fc68d623d1bfc36c8cb2fe4a71f19c8578cfb420ce8ce07b20a02c1ee0be0cf3 [e2e-llm-inference-service] kserve-llm-d-inference-scheduler=quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] kserve-llm-d-routing-sidecar=quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.9.0 [e2e-llm-inference-service] kserve-llm-d-uds-tokenizer=quay.io/opendatahub/llm-d-kv-cache:v0.8.0 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda=registry.redhat.io/rhaiis/vllm-cuda-rhel9@sha256:fc68d623d1bfc36c8cb2fe4a71f19c8578cfb420ce8ce07b20a02c1ee0be0cf3 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda-fast-1=registry.redhat.io/rhaiis/vllm-cuda-rhel9@sha256:fc68d623d1bfc36c8cb2fe4a71f19c8578cfb420ce8ce07b20a02c1ee0be0cf3 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda-fast-2=registry.redhat.io/rhaiis/vllm-cuda-rhel9@sha256:fc68d623d1bfc36c8cb2fe4a71f19c8578cfb420ce8ce07b20a02c1ee0be0cf3 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda-fast-1-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda-fast-2-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-amd-rocm=registry.redhat.io/rhaiis/vllm-rocm-rhel9@sha256:d9a48add238cc095fa43eeee17c8c4d104de60c4dc623e0bc7f8c4b53b2b2e97 [e2e-llm-inference-service] kserve-llm-d-amd-rocm-fast-1=registry.redhat.io/rhaiis/vllm-rocm-rhel9@sha256:d9a48add238cc095fa43eeee17c8c4d104de60c4dc623e0bc7f8c4b53b2b2e97 [e2e-llm-inference-service] kserve-llm-d-amd-rocm-fast-2=registry.redhat.io/rhaiis/vllm-rocm-rhel9@sha256:d9a48add238cc095fa43eeee17c8c4d104de60c4dc623e0bc7f8c4b53b2b2e97 [e2e-llm-inference-service] kserve-llm-d-amd-rocm-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-amd-rocm-fast-1-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-amd-rocm-fast-2-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi=registry.redhat.io/rhaii-early-access/vllm-gaudi-rhel9:3.4.0-ea.2 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi-fast-1=registry.redhat.io/rhaii-early-access/vllm-gaudi-rhel9:3.4.0-ea.2 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi-fast-2=registry.redhat.io/rhaii-early-access/vllm-gaudi-rhel9:3.4.0-ea.2 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi-upstream-version=0.16.0 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi-fast-1-upstream-version=0.16.0 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi-fast-2-upstream-version=0.16.0 [e2e-llm-inference-service] kserve-llm-d-ibm-spyre=registry.redhat.io/rhaiis/vllm-spyre-rhel9@sha256:80ae3e435a5be2c1f117f36599103ab05357917dd6e37f0df6613cb3ac2c13ea [e2e-llm-inference-service] kserve-llm-d-ibm-spyre-fast-1=registry.redhat.io/rhaiis/vllm-spyre-rhel9@sha256:80ae3e435a5be2c1f117f36599103ab05357917dd6e37f0df6613cb3ac2c13ea [e2e-llm-inference-service] kserve-llm-d-ibm-spyre-fast-2=registry.redhat.io/rhaiis/vllm-spyre-rhel9@sha256:80ae3e435a5be2c1f117f36599103ab05357917dd6e37f0df6613cb3ac2c13ea [e2e-llm-inference-service] kserve-llm-d-ibm-spyre-upstream-version=0.10.2 [e2e-llm-inference-service] kserve-llm-d-ibm-spyre-fast-1-upstream-version=0.10.2 [e2e-llm-inference-service] kserve-llm-d-ibm-spyre-fast-2-upstream-version=0.10.2 [e2e-llm-inference-service] kserve-llm-d-latency-predictor-prediction=quay.io/opendatahub/odh-latency-predictor-prediction:odh-stable [e2e-llm-inference-service] kserve-llm-d-latency-predictor-training=quay.io/opendatahub/odh-latency-predictor-training:odh-stable [e2e-llm-inference-service] # TODO update when our changes are introduced in the official image [e2e-llm-inference-service] kube-rbac-proxy=quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3 [e2e-llm-inference-service] kserve-localmodel-controller=quay.io/opendatahub/odh-kserve-localmodel-controller:odh-stable [e2e-llm-inference-service] kserve-localmodelnode-agent=quay.io/opendatahub/odh-kserve-localmodelnode-agent:odh-stable [e2e-llm-inference-service] ovms-versioning-ubi-micro=registry.redhat.io/ubi9/ubi-micro@sha256:38e934147827349f2b8b11ac9c38d7be23cb8e29f128b1c46e5c7ae54a2d23cd [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/dscinitializations.dscinitialization.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencemodelrewrites.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceobjectives.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-llm-inference-service] Waiting for CRDs to be established... [e2e-llm-inference-service] Waiting for CRD inferenceservices.serving.kserve.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD inferenceservices.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io condition met [e2e-llm-inference-service] Waiting for CRD llminferenceserviceconfigs.serving.kserve.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD llminferenceserviceconfigs.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io condition met [e2e-llm-inference-service] Waiting for CRD clusterstoragecontainers.serving.kserve.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD clusterstoragecontainers.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io condition met [e2e-llm-inference-service] Waiting for CRD datascienceclusters.datasciencecluster.opendatahub.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD datascienceclusters.datasciencecluster.opendatahub.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io condition met [e2e-llm-inference-service] Applying all resources... [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/dscinitializations.dscinitialization.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencemodelrewrites.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceobjectives.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-llm-inference-service] serviceaccount/kserve-controller-manager serverside-applied [e2e-llm-inference-service] serviceaccount/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/kserve-leader-election-role serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/kserve-llmisvcconfig-read-access serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/llmisvc-leader-election-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-admin serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-edit serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-inferenceservice-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-manager-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-manager-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-metrics-reader-cluster-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-models-admin serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-models-edit serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-models-view serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-proxy-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-tls-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-view serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/openshift-ai-inferenceservice-image-volume-scc serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/kserve-leader-election-rolebinding serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/kserve-llmisvcconfig-read-access serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/llmisvc-leader-election-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-inferenceservice-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-llmisvc-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-manager-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-proxy-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-tls-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/llmisvc-manager-rolebinding serverside-applied [e2e-llm-inference-service] configmap/inferenceservice-config serverside-applied [e2e-llm-inference-service] configmap/kserve-parameters serverside-applied [e2e-llm-inference-service] secret/kserve-webhook-server-secret serverside-applied [e2e-llm-inference-service] secret/mlpipeline-s3-artifact serverside-applied [e2e-llm-inference-service] service/kserve-controller-manager-metrics-service serverside-applied [e2e-llm-inference-service] service/kserve-controller-manager-service serverside-applied [e2e-llm-inference-service] service/kserve-webhook-server-service serverside-applied [e2e-llm-inference-service] service/llmisvc-controller-manager-service serverside-applied [e2e-llm-inference-service] service/llmisvc-webhook-server-service serverside-applied [e2e-llm-inference-service] service/s3-service serverside-applied [e2e-llm-inference-service] deployment.apps/kserve-controller-manager serverside-applied [e2e-llm-inference-service] deployment.apps/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] deployment.apps/seaweedfs serverside-applied [e2e-llm-inference-service] networkpolicy.networking.k8s.io/kserve-controller-manager serverside-applied [e2e-llm-inference-service] securitycontextconstraints.security.openshift.io/openshift-ai-inferenceservice-image-volume-scc serverside-applied [e2e-llm-inference-service] securitycontextconstraints.security.openshift.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-llm-inference-service] clusterstoragecontainer.serving.kserve.io/default serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-router-route serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler-latency-predictor serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-tokenizer serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-tracing serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-worker-data-parallel serverside-applied [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/inferencegraph.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceserviceconfig.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/servingruntime.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/trainedmodel.serving.kserve.io serverside-applied [e2e-llm-inference-service] Waiting for llmisvc-controller-manager to be ready... [e2e-llm-inference-service] Waiting for pod -l "control-plane=llmisvc-controller-manager" in namespace "kserve" to be created... [e2e-llm-inference-service] Pod -l "control-plane=llmisvc-controller-manager" in namespace "kserve" found. [e2e-llm-inference-service] Current pods for -l "control-plane=llmisvc-controller-manager" in namespace "kserve": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] llmisvc-controller-manager-67c47db88-fw5qc 0/1 Running 0 6s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "control-plane=llmisvc-controller-manager" in namespace "kserve" to become ready... [e2e-llm-inference-service] pod/llmisvc-controller-manager-67c47db88-fw5qc condition met [e2e-llm-inference-service] Pod(s) -l "control-plane=llmisvc-controller-manager" in namespace "kserve" are ready. [e2e-llm-inference-service] Re-Applying all resources... [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/dscinitializations.dscinitialization.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencemodelrewrites.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceobjectives.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-llm-inference-service] serviceaccount/kserve-controller-manager serverside-applied [e2e-llm-inference-service] serviceaccount/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/kserve-leader-election-role serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/kserve-llmisvcconfig-read-access serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/llmisvc-leader-election-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-admin serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-edit serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-inferenceservice-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-manager-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-manager-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-metrics-reader-cluster-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-models-admin serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-models-edit serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-models-view serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-proxy-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-tls-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-view serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/openshift-ai-inferenceservice-image-volume-scc serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/kserve-leader-election-rolebinding serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/kserve-llmisvcconfig-read-access serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/llmisvc-leader-election-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-inferenceservice-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-llmisvc-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-manager-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-proxy-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-tls-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/llmisvc-manager-rolebinding serverside-applied [e2e-llm-inference-service] configmap/inferenceservice-config serverside-applied [e2e-llm-inference-service] configmap/kserve-parameters serverside-applied [e2e-llm-inference-service] secret/kserve-webhook-server-secret serverside-applied [e2e-llm-inference-service] secret/mlpipeline-s3-artifact serverside-applied [e2e-llm-inference-service] service/kserve-controller-manager-metrics-service serverside-applied [e2e-llm-inference-service] service/kserve-controller-manager-service serverside-applied [e2e-llm-inference-service] service/kserve-webhook-server-service serverside-applied [e2e-llm-inference-service] service/llmisvc-controller-manager-service serverside-applied [e2e-llm-inference-service] service/llmisvc-webhook-server-service serverside-applied [e2e-llm-inference-service] service/s3-service serverside-applied [e2e-llm-inference-service] deployment.apps/kserve-controller-manager serverside-applied [e2e-llm-inference-service] deployment.apps/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] deployment.apps/seaweedfs serverside-applied [e2e-llm-inference-service] networkpolicy.networking.k8s.io/kserve-controller-manager serverside-applied [e2e-llm-inference-service] securitycontextconstraints.security.openshift.io/openshift-ai-inferenceservice-image-volume-scc serverside-applied [e2e-llm-inference-service] securitycontextconstraints.security.openshift.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-llm-inference-service] clusterstoragecontainer.serving.kserve.io/default serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-router-route serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler-latency-predictor serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-tokenizer serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-tracing serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-worker-data-parallel serverside-applied [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/inferencegraph.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceserviceconfig.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/servingruntime.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/trainedmodel.serving.kserve.io serverside-applied [e2e-llm-inference-service] Applying DSC/DSCI resources... [e2e-llm-inference-service] dscinitialization.dscinitialization.opendatahub.io/test-dsci created [e2e-llm-inference-service] datasciencecluster.datasciencecluster.opendatahub.io/test-dsc created [e2e-llm-inference-service] KServe manual installation complete [e2e-llm-inference-service] 🔧 Configuration: [e2e-llm-inference-service] KServe deployment: ❌ disabled [e2e-llm-inference-service] Kuadrant deployment: ✅ enabled [e2e-llm-inference-service] [e2e-llm-inference-service] Checking OpenShift server version...(4.21.26) [e2e-llm-inference-service] 🎯 Server version (4.21.26) is 4.19.9 or higher - continue with the script [e2e-llm-inference-service] ⏳ Installing cert-manager [e2e-llm-inference-service] namespace/cert-manager-operator created [e2e-llm-inference-service] operatorgroup.operators.coreos.com/openshift-cert-manager-operator created [e2e-llm-inference-service] subscription.operators.coreos.com/openshift-cert-manager-operator created [e2e-llm-inference-service] Waiting for openshift-cert-manager-operator CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (0/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (5/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (10/300) [e2e-llm-inference-service] CSV cert-manager-operator.v1.20.0 found, but not yet Succeeded (Phase: Installing). Waiting... (15/300) [e2e-llm-inference-service] CSV cert-manager-operator.v1.20.0 is ready (Phase: Succeeded). [e2e-llm-inference-service] Waiting for CRD certificates.cert-manager.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD certificates.cert-manager.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io condition met [e2e-llm-inference-service] ✅ Cert-manager installed [e2e-llm-inference-service] ⏳ Installing openshift-lws-operator [e2e-llm-inference-service] namespace/openshift-lws-operator created [e2e-llm-inference-service] operatorgroup.operators.coreos.com/leader-worker-set created [e2e-llm-inference-service] subscription.operators.coreos.com/leader-worker-set created [e2e-llm-inference-service] Waiting for leader-worker-set CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (0/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (5/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (10/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (15/300) [e2e-llm-inference-service] CSV leader-worker-set.v1.0.0 is ready (Phase: Succeeded). [e2e-llm-inference-service] Waiting for CRD leaderworkersetoperators.operator.openshift.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD leaderworkersetoperators.operator.openshift.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/leaderworkersetoperators.operator.openshift.io condition met [e2e-llm-inference-service] leaderworkersetoperator.operator.openshift.io/cluster created [e2e-llm-inference-service] ⏳ waiting for openshift-lws-operator to be ready.… [e2e-llm-inference-service] Waiting for pod -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" to be created... [e2e-llm-inference-service] Pod -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" found. [e2e-llm-inference-service] Current pods for -l "name=openshift-lws-operator" in namespace "openshift-lws-operator": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] openshift-lws-operator-fd8ccff4c-rd9zg 1/1 Running 0 13s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" to become ready... [e2e-llm-inference-service] pod/openshift-lws-operator-fd8ccff4c-rd9zg condition met [e2e-llm-inference-service] Pod(s) -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" are ready. [e2e-llm-inference-service] ✅ openshift-lws-operator installed [e2e-llm-inference-service] gatewayclass.gateway.networking.k8s.io/openshift-default created [e2e-llm-inference-service] Waiting for pod -l "app=istiod" in namespace "openshift-ingress" to be created... [e2e-llm-inference-service] Pod -l "app=istiod" in namespace "openshift-ingress" found. [e2e-llm-inference-service] Current pods for -l "app=istiod" in namespace "openshift-ingress": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] istiod-openshift-gateway-5677ccb49b-46d7p 1/1 Running 0 6s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "app=istiod" in namespace "openshift-ingress" to become ready... [e2e-llm-inference-service] pod/istiod-openshift-gateway-5677ccb49b-46d7p condition met [e2e-llm-inference-service] Pod(s) -l "app=istiod" in namespace "openshift-ingress" are ready. [e2e-llm-inference-service] ⏳ Creating a Gateway [e2e-llm-inference-service] Error from server (AlreadyExists): namespaces "openshift-ingress" already exists [e2e-llm-inference-service] ⏳ Creating gateway memory ConfigMap for parametersRef (2Gi) [e2e-llm-inference-service] configmap/gateway-proxy-config created [e2e-llm-inference-service] gateway.gateway.networking.k8s.io/openshift-ai-inference created [e2e-llm-inference-service] Waiting for pod -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" to be created... [e2e-llm-inference-service] Pod -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" found. [e2e-llm-inference-service] Current pods for -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] openshift-ai-inference-openshift-default-8699b59bbc-xf9js 1/1 Running 0 5s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" to become ready... [e2e-llm-inference-service] pod/openshift-ai-inference-openshift-default-8699b59bbc-xf9js condition met [e2e-llm-inference-service] Pod(s) -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" are ready. [e2e-llm-inference-service] ⏳ Installing RHCL(Kuadrant) operator [e2e-llm-inference-service] namespace/kuadrant-system created [e2e-llm-inference-service] subscription.operators.coreos.com/rhcl-operator created [e2e-llm-inference-service] operatorgroup.operators.coreos.com/kuadrant created [e2e-llm-inference-service] Waiting for rhcl-operator CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (0/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (5/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (10/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (15/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (20/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (25/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.2 found, but not yet Succeeded (Phase: Installing). Waiting... (30/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.2 found, but not yet Succeeded (Phase: Installing). Waiting... (35/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.2 found, but not yet Succeeded (Phase: Installing). Waiting... (40/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.2 is ready (Phase: Succeeded). [e2e-llm-inference-service] Waiting for CRD kuadrants.kuadrant.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD kuadrants.kuadrant.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/kuadrants.kuadrant.io condition met [e2e-llm-inference-service] Waiting for apiserver discovery /apis/kuadrant.io/v1beta1 to list kuadrants (timeout: 120s)… [e2e-llm-inference-service] Discovery for kuadrant.io/v1beta1 includes kuadrants. [e2e-llm-inference-service] ⏳ sleeping 30s after discovery (RESTMapper can trail discovery)… [e2e-llm-inference-service] kuadrant.kuadrant.io/kuadrant created [e2e-llm-inference-service] ⏳ waiting for Kuadrant Ready (attempt 1/2, timeout 5m)… [e2e-llm-inference-service] kuadrant.kuadrant.io/kuadrant condition met [e2e-llm-inference-service] Waiting for pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" to be created... [e2e-llm-inference-service] Pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" found. [e2e-llm-inference-service] Current pods for -l "control-plane=authorino-operator" in namespace "kuadrant-system": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] authorino-operator-56fb6f6856-rzd9v 1/1 Running 0 67s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" to become ready... [e2e-llm-inference-service] pod/authorino-operator-56fb6f6856-rzd9v condition met [e2e-llm-inference-service] Pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" are ready. [e2e-llm-inference-service] ⏳ waiting for authorino service to be created... [e2e-llm-inference-service] service/authorino-authorino-authorization condition met [e2e-llm-inference-service] service/authorino-authorino-authorization annotated [e2e-llm-inference-service] Warning: resource authorinos/authorino is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by oc apply. oc apply should only be used on resources created declaratively by either oc create --save-config or oc apply. The missing annotation will be patched automatically. [e2e-llm-inference-service] authorino.operator.authorino.kuadrant.io/authorino configured [e2e-llm-inference-service] Waiting for pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" to be created... [e2e-llm-inference-service] Pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" found. [e2e-llm-inference-service] Current pods for -l "control-plane=authorino-operator" in namespace "kuadrant-system": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] authorino-operator-56fb6f6856-rzd9v 1/1 Running 0 77s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" to become ready... [e2e-llm-inference-service] pod/authorino-operator-56fb6f6856-rzd9v condition met [e2e-llm-inference-service] Pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" are ready. [e2e-llm-inference-service] ✅ kuadrant(authorino) installed [e2e-llm-inference-service] Setting up Jaeger (Helm) for LLMISVC tracing e2e... [e2e-llm-inference-service] ⏳ Installing Jaeger All-in-One (Helm) into namespace observability [e2e-llm-inference-service] [INFO] Installing Helm v3.16.3 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed Helm v3.16.3 to /workspace/source/bin/helm [e2e-llm-inference-service] WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /credentials/cluster-zm6s9-kubeconfig [e2e-llm-inference-service] WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /credentials/cluster-zm6s9-kubeconfig [e2e-llm-inference-service] version.BuildInfo{Version:"v3.16.3", GitCommit:"cfd07493f46efc9debd9cc1b02a0961186df7fdf", GitTreeState:"clean", GoVersion:"go1.22.7"} [e2e-llm-inference-service] [INFO] Adding Jaeger Helm repository... [e2e-llm-inference-service] WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /credentials/cluster-zm6s9-kubeconfig [e2e-llm-inference-service] WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /credentials/cluster-zm6s9-kubeconfig [e2e-llm-inference-service] "jaegertracing" has been added to your repositories [e2e-llm-inference-service] [INFO] Installing Jaeger All-in-One 4.7.0... [e2e-llm-inference-service] WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /credentials/cluster-zm6s9-kubeconfig [e2e-llm-inference-service] WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /credentials/cluster-zm6s9-kubeconfig [e2e-llm-inference-service] NAME: jaeger [e2e-llm-inference-service] LAST DEPLOYED: Wed Jul 29 10:38:58 2026 [e2e-llm-inference-service] NAMESPACE: observability [e2e-llm-inference-service] STATUS: deployed [e2e-llm-inference-service] REVISION: 1 [e2e-llm-inference-service] TEST SUITE: None [e2e-llm-inference-service] NOTES: [e2e-llm-inference-service] ################################################################### [e2e-llm-inference-service] ### ⚠️ EXPERIMENTAL - NO STABILITY GUARANTEES ### [e2e-llm-inference-service] ### ### [e2e-llm-inference-service] ### This chart is under active development. ### [e2e-llm-inference-service] ### Breaking changes may occur in minor versions. ### [e2e-llm-inference-service] ### ### [e2e-llm-inference-service] ### See README.md for configuration details. ### [e2e-llm-inference-service] ################################################################### [e2e-llm-inference-service] [e2e-llm-inference-service] 🚀 Congratulations on successfully installing Jaeger v2.17.0 (Chart v4.7.0)! [e2e-llm-inference-service] [e2e-llm-inference-service] To access the query UI: [e2e-llm-inference-service] export POD_NAME=$(kubectl get pods --namespace observability -l "app.kubernetes.io/instance=jaeger,app.kubernetes.io/component=all-in-one" -o jsonpath="{.items[0].metadata.name}") [e2e-llm-inference-service] kubectl port-forward --namespace observability $POD_NAME 16686:16686 --address 0.0.0.0 & [e2e-llm-inference-service] Visit http://127.0.0.1:16686/ [e2e-llm-inference-service] [SUCCESS] Successfully installed Jaeger All-in-One via Helm [e2e-llm-inference-service] [INFO] Waiting for pods with label 'app.kubernetes.io/name=jaeger' in namespace 'observability' to be created... [e2e-llm-inference-service] [INFO] Found 1 pod(s) with label 'app.kubernetes.io/name=jaeger' [e2e-llm-inference-service] [INFO] Waiting for pods with label 'app.kubernetes.io/name=jaeger' in namespace 'observability' to be ready... [e2e-llm-inference-service] pod/jaeger-6bbd7d4c9d-2rv7j condition met [e2e-llm-inference-service] [SUCCESS] Pods with label 'app.kubernetes.io/name=jaeger' in namespace 'observability' are ready! [e2e-llm-inference-service] [SUCCESS] Jaeger is ready! [e2e-llm-inference-service] ⏳ Verifying Jaeger Service ports (OTLP 4317, Query 16686)... [e2e-llm-inference-service] ✅ Jaeger (Helm) ready — OTLP http://jaeger.observability.svc.cluster.local:4317 [e2e-llm-inference-service] Patching ingress domain... [e2e-llm-inference-service] configmap/inferenceservice-config patched [e2e-llm-inference-service] pod "kserve-controller-manager-65f566c74b-v6w4s" deleted [e2e-llm-inference-service] Waiting for kserve-controller-manager to be ready... [e2e-llm-inference-service] pod/kserve-controller-manager-65f566c74b-m6f8j condition met [e2e-llm-inference-service] Installing ODH Model Controller manually... [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/accounts.nim.opendatahub.io created [e2e-llm-inference-service] serviceaccount/model-serving-api created [e2e-llm-inference-service] serviceaccount/odh-model-controller created [e2e-llm-inference-service] role.rbac.authorization.k8s.io/leader-election-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/account-editor-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/account-viewer-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-prometheus-k8s created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/metrics-reader created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/model-serving-api created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/odh-model-controller-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/proxy-role created [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/leader-election-rolebinding created [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/model-serving-api created [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/odh-model-controller-rolebinding-opendatahub created [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/proxy-rolebinding created [e2e-llm-inference-service] configmap/odh-model-controller-parameters created [e2e-llm-inference-service] service/model-serving-api created [e2e-llm-inference-service] service/odh-model-controller-metrics-service created [e2e-llm-inference-service] service/odh-model-controller-webhook-service created [e2e-llm-inference-service] deployment.apps/model-serving-api created [e2e-llm-inference-service] deployment.apps/odh-model-controller created [e2e-llm-inference-service] servicemonitor.monitoring.coreos.com/model-serving-api-metrics created [e2e-llm-inference-service] servicemonitor.monitoring.coreos.com/odh-model-controller-metrics-monitor created [e2e-llm-inference-service] template.template.openshift.io/autogluon-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/guardrails-detector-huggingface-serving-template created [e2e-llm-inference-service] template.template.openshift.io/kserve-ovms created [e2e-llm-inference-service] template.template.openshift.io/mlserver-cuda-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/mlserver-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-x86-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-x86-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-x86-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cuda-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cuda-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cuda-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-gaudi-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-gaudi-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-gaudi-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-multinode-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-multinode-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-multinode-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-rocm-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-rocm-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-rocm-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-ppc64le-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-ppc64le-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-ppc64le-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-s390x-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-s390x-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-s390x-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-x86-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-x86-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-x86-runtime-template-fast-2 created [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/mutating.odh-model-controller.opendatahub.io created [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/validating.odh-model-controller.opendatahub.io created [e2e-llm-inference-service] Waiting for deployment "odh-model-controller" rollout to finish: 0 of 1 updated replicas are available... [e2e-llm-inference-service] deployment "odh-model-controller" successfully rolled out [e2e-llm-inference-service] networkpolicy.networking.k8s.io/allow-all created [e2e-llm-inference-service] KServe setup complete (namespace: kserve) [e2e-llm-inference-service] Add testing models to SeaweedFS S3 storage ... [e2e-llm-inference-service] Waiting for SeaweedFS deployment to be ready... [e2e-llm-inference-service] deployment "seaweedfs" successfully rolled out [e2e-llm-inference-service] S3 init job not completed, re-creating... [e2e-llm-inference-service] job.batch/s3-init replaced [e2e-llm-inference-service] Waiting for S3 init job to complete... [e2e-llm-inference-service] job.batch/s3-init condition met [e2e-llm-inference-service] Prepare CI namespace and install ServingRuntimes [e2e-llm-inference-service] Setting up CI namespace: kserve-ci-e2e-test [e2e-llm-inference-service] Tearing down CI namespace: kserve-ci-e2e-test [e2e-llm-inference-service] Namespace kserve-ci-e2e-test does not exist, skipping deletion [e2e-llm-inference-service] CI namespace teardown complete [e2e-llm-inference-service] Creating namespace kserve-ci-e2e-test [e2e-llm-inference-service] namespace/kserve-ci-e2e-test created [e2e-llm-inference-service] Applying S3 artifact secret [e2e-llm-inference-service] secret/mlpipeline-s3-artifact created [e2e-llm-inference-service] Applying storage-config secret [e2e-llm-inference-service] secret/storage-config created [e2e-llm-inference-service] Applying SeaweedFS S3 credentials secret [e2e-llm-inference-service] secret/seaweedfs-s3-creds created [e2e-llm-inference-service] Linking seaweedfs-s3-creds to default service account [e2e-llm-inference-service] Creating odh-trusted-ca-bundle configmap [e2e-llm-inference-service] configmap/odh-trusted-ca-bundle created [e2e-llm-inference-service] Installing ServingRuntimes [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-autogluonserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-huggingfaceserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-huggingfaceserver-multinode created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-lgbserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-mlserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-paddleserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-pmmlserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-predictiveserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-sklearnserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-tensorflow-serving created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-torchserve created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-tritonserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-vllmserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-xgbserver created [e2e-llm-inference-service] CI namespace setup complete [e2e-llm-inference-service] Setup complete [e2e-llm-inference-service] === E2E cluster / operator summary === [e2e-llm-inference-service] Client Version: 4.20.11 [e2e-llm-inference-service] Kustomize Version: v5.6.0 [e2e-llm-inference-service] Server Version: 4.21.26 [e2e-llm-inference-service] Kubernetes Version: v1.34.9 [e2e-llm-inference-service] ClusterVersion desired: 4.21.26 [e2e-llm-inference-service] ClusterVersion history (latest): 4.21.26 (Completed) [e2e-llm-inference-service] CSVs in kuadrant-system: [e2e-llm-inference-service] authorino-operator.v1.4.2 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.0 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.1 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.1 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.2 Succeeded [e2e-llm-inference-service] CSVs in openshift-keda: [e2e-llm-inference-service] authorino-operator.v1.4.2 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.0 Succeeded [e2e-llm-inference-service] custom-metrics-autoscaler.v2.19.0-2 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.1 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.1 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.2 Succeeded [e2e-llm-inference-service] CSVs in cert-manager-operator: [e2e-llm-inference-service] authorino-operator.v1.4.2 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.0 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.1 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.1 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.2 Succeeded [e2e-llm-inference-service] CSVs in openshift-lws-operator: [e2e-llm-inference-service] authorino-operator.v1.4.2 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.0 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.1 Succeeded [e2e-llm-inference-service] leader-worker-set.v1.0.0 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.1 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.2 Succeeded [e2e-llm-inference-service] CSVs in openshift-operators (ODH / shared operators, filtered): [e2e-llm-inference-service] authorino-operator.v1.4.2 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.1 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.1 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.2 Succeeded [e2e-llm-inference-service] Kuadrant / Authorino (diagnostics): [e2e-llm-inference-service] CRD kuadrants.kuadrant.io versions: v1beta1 served=true storage=true [e2e-llm-inference-service] Subscriptions in kuadrant-system: [e2e-llm-inference-service] authorino-operator-stable-redhat-operators-openshift-marketplace stable redhat-operators authorino-operator.v1.4.2 [e2e-llm-inference-service] dns-operator-stable-redhat-operators-openshift-marketplace stable redhat-operators dns-operator.v1.4.1 [e2e-llm-inference-service] limitador-operator-stable-redhat-operators-openshift-marketplace stable redhat-operators limitador-operator.v1.4.1 [e2e-llm-inference-service] rhcl-operator stable redhat-operators rhcl-operator.v1.4.2 [e2e-llm-inference-service] Kuadrant CR conditions (kuadrant/kuadrant-system): [e2e-llm-inference-service] Ready=True (Ready) [e2e-llm-inference-service] KServe deployments in kserve: [e2e-llm-inference-service] kserve-controller-manager: ready=1 image=quay.io/opendatahub/kserve-controller@sha256:6dbe90fc1e66fa697d36172c78c604df1b033c60023ad052bada7adb20987c06 [e2e-llm-inference-service] imageID: quay.io/opendatahub/kserve-controller@sha256:0750b32843ab936357317dd414decba5b3fb78bfa13f8c1841e1a03758619b34 [e2e-llm-inference-service] odh-model-controller: ready=1 image=quay.io/opendatahub/odh-model-controller:odh-incubating [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-model-controller@sha256:db7c48ffe2308ebb296fbfa3f952415e49a1ca6943ea95d9d0748b2d8b7d7770 [e2e-llm-inference-service] llmisvc-controller-manager: ready=1 image=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:c6eafbd6173b198967c1d458a4b4bb056f2d32a2d9c083284e09b22ad9a1149a [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:672643964335a6a56f347ba6da24254964155e3f1276061ecc5e0122c0a5fa19 [e2e-llm-inference-service] === End E2E cluster / operator summary === [e2e-llm-inference-service] /workspace/source [e2e-llm-inference-service] CA certificate extracted [e2e-llm-inference-service] REQUESTS_CA_BUNDLE=/tmp/ca.crt [e2e-llm-inference-service] Run E2E tests: llminferenceservice and cluster_cpu and not autoscaling and not tracing [e2e-llm-inference-service] Starting E2E functional tests ... [e2e-llm-inference-service] Parallelism requested for pytest is 2 [e2e-llm-inference-service] ============================= test session starts ============================== [e2e-llm-inference-service] platform linux -- Python 3.11.13, pytest-7.4.4, pluggy-1.5.0 -- /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] cachedir: .pytest_cache [e2e-llm-inference-service] metadata: {'Python': '3.11.13', 'Platform': 'Linux-5.14.0-570.124.1.el9_6.x86_64-x86_64-with-glibc2.34', 'Packages': {'pytest': '7.4.4', 'pluggy': '1.5.0'}, 'Plugins': {'metadata': '3.1.1', 'anyio': '4.9.0', 'xdist': '3.6.1', 'json-report': '1.5.0', 'httpx': '0.30.0', 'cov': '5.0.0', 'asyncio': '0.23.8'}, 'PLATFORM': 'el9'} [e2e-llm-inference-service] rootdir: /workspace/source/test/e2e [e2e-llm-inference-service] configfile: pytest.ini [e2e-llm-inference-service] plugins: metadata-3.1.1, anyio-4.9.0, xdist-3.6.1, json-report-1.5.0, httpx-0.30.0, cov-5.0.0, asyncio-0.23.8 [e2e-llm-inference-service] asyncio: mode=Mode.STRICT [e2e-llm-inference-service] created: 2/2 workers [e2e-llm-inference-service] 2 workers [73 items] [e2e-llm-inference-service] [e2e-llm-inference-service] scheduling tests via WorkStealingScheduling [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-utilization-detector] 2026-07-29 10:41:25.747 6681 kserve INFO [conftest.py:configure_logger():40] Logger configured [e2e-llm-inference-service] 2026-07-29 10:41:25.748 6684 kserve INFO [conftest.py:configure_logger():40] Logger configured [e2e-llm-inference-service] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-utilization-detector] [e2e-llm-inference-service] llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-concurrency-detector] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-concurrency-detector] [e2e-llm-inference-service] llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-with-section-name] 2026-07-29 10:43:41.473 6684 kserve.trace Checking Gateway router-gateway-1 in namespace e2e-test-gateway-section-name-propagation-49b5edd9 [e2e-llm-inference-service] 2026-07-29 10:43:41.473 6684 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-1 in namespace e2e-test-gateway-section-name-propagation-49b5edd9 [e2e-llm-inference-service] 2026-07-29 10:43:41.497 6684 kserve.trace Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-29 10:43:41.497 6684 kserve.trace INFO [gw_api.py:create_or_update_gateway():62] Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-29 10:43:41.507 6684 kserve.trace ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-29 10:43:41.507 6684 kserve.trace INFO [gw_api.py:create_or_update_gateway():70] ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-with-section-name] [e2e-llm-inference-service] llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-without-section-name] 2026-07-29 10:43:54.156 6684 kserve.trace Checking Gateway router-gateway-1 in namespace e2e-test-gateway-section-name-propagation-94799d44 [e2e-llm-inference-service] 2026-07-29 10:43:54.156 6684 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-1 in namespace e2e-test-gateway-section-name-propagation-94799d44 [e2e-llm-inference-service] 2026-07-29 10:43:54.188 6684 kserve.trace Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-29 10:43:54.188 6684 kserve.trace INFO [gw_api.py:create_or_update_gateway():62] Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-29 10:43:54.207 6684 kserve.trace ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-29 10:43:54.207 6684 kserve.trace INFO [gw_api.py:create_or_update_gateway():70] ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-without-section-name] [e2e-llm-inference-service] llmisvc/test_llm_auth.py::test_llm_auth_enabled_requires_token[cluster_cpu-cluster_single_node-auth-enabled-default] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_auth.py::test_llm_auth_enabled_requires_token[cluster_cpu-cluster_single_node-auth-enabled-default] [e2e-llm-inference-service] llmisvc/test_llm_auth.py::test_llm_auth_invalid_token_rejected[cluster_cpu-cluster_single_node-auth-invalid-token] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-inline-config-workload-llmd-simulator] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-inline-config-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-qwen2.5-0.5b] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-qwen2.5-0.5b] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-configmap-ref-workload-llmd-simulator] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_auth.py::test_llm_auth_invalid_token_rejected[cluster_cpu-cluster_single_node-auth-invalid-token] [e2e-llm-inference-service] llmisvc/test_llm_auth.py::test_llm_auth_disabled_no_token_required[cluster_cpu-cluster_single_node-auth-disabled] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-configmap-ref-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-replicas-workload-llmd-simulator] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-replicas-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-custom-template-workload-llmd-simulator] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_auth.py::test_llm_auth_disabled_no_token_required[cluster_cpu-cluster_single_node-auth-disabled] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_hpa_deployment[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-custom-template-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-nonzero-threshold-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-nonzero-threshold-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-tokenizer-kvcache-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-tokenizer-kvcache-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator0] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator1] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator2] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator2] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf0] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] [e2e-llm-inference-service] [gw1] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_hpa_deployment[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_keda_deployment[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-keda] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-pvc] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-pvc] [e2e-llm-inference-service] [gw1] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_keda_deployment[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-keda] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_hpa_lws[cluster_cpu-cluster_multi_node-router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] [gw0] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-pvc] [e2e-llm-inference-service] [gw1] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_hpa_lws[cluster_cpu-cluster_multi_node-router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_keda_lws[cluster_cpu-cluster_multi_node-router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-keda] [e2e-llm-inference-service] [gw0] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service_config_deletion.py::test_config_finalizer_added [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service_config_deletion.py::test_config_finalizer_added [e2e-llm-inference-service] llmisvc/test_llm_inference_service_config_deletion.py::test_config_deletion_blocked_when_referenced [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service_config_deletion.py::test_config_deletion_blocked_when_referenced [e2e-llm-inference-service] llmisvc/test_llm_inference_service_config_deletion.py::test_config_deletion_allowed_when_unreferenced [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service_config_deletion.py::test_config_deletion_allowed_when_unreferenced [e2e-llm-inference-service] llmisvc/test_llm_inference_service_config_deletion.py::test_config_deletion_unblocked_after_service_deleted [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service_config_deletion.py::test_config_deletion_unblocked_after_service_deleted [e2e-llm-inference-service] llmisvc/test_llm_inference_service_config_deletion.py::test_well_known_config_deletion_prevented_by_webhook [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service_config_deletion.py::test_well_known_config_deletion_prevented_by_webhook [e2e-llm-inference-service] llmisvc/test_llm_inference_service_config_deletion.py::test_well_known_config_deletion_blocked_by_implicit_reference [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service_config_deletion.py::test_well_known_config_deletion_blocked_by_implicit_reference [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha1_to_v1alpha2_conversion [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha1_to_v1alpha2_conversion [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha2_to_v1alpha1_conversion [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha2_to_v1alpha1_conversion [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_criticality_preservation_via_annotations [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_criticality_preservation_via_annotations [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_lora_criticality_preservation [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_lora_criticality_preservation [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_round_trip_conversion_preserves_fields [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_round_trip_conversion_preserves_fields [e2e-llm-inference-service] llmisvc/test_llm_inference_service_stop.py::test_llm_stop_feature[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service_stop.py::test_llm_stop_feature[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-single-lora-adapter-hf] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-single-lora-adapter-hf] [e2e-llm-inference-service] llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-multiple-lora-adapters] [e2e-llm-inference-service] [gw1] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_keda_lws[cluster_cpu-cluster_multi_node-router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-keda] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_cleanup_hpa[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-multiple-lora-adapters] [e2e-llm-inference-service] llmisvc/test_llm_tls.py::test_llm_tls_resources[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_tls.py::test_llm_tls_resources[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_prestop_hook.py::test_prestop_hook[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_prestop_hook.py::test_prestop_hook[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_rolling_upgrade.py::test_rolling_upgrade_coordination[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_rolling_upgrade.py::test_rolling_upgrade_coordination[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_storage_version_migration.py::TestStorageVersionMigration::test_storage_version_migration_after_simulated_upgrade [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_storage_version_migration.py::TestStorageVersionMigration::test_storage_version_migration_after_simulated_upgrade [e2e-llm-inference-service] [gw1] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_cleanup_hpa[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] [e2e-llm-inference-service] =================================== FAILURES =================================== [e2e-llm-inference-service] _ test_llm_autoscaling_hpa_deployment[router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-hpa] _ [e2e-llm-inference-service] [gw1] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-hpa'], prom... {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.autoscaling_hpa [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator-no-replicas", [e2e-llm-inference-service] "prometheus-scrape", [e2e-llm-inference-service] "scaling-hpa", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="autoscale-hpa-deploy", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_autoscaling_hpa_deployment(test_case: TestCase): [e2e-llm-inference-service] """HPA + Deployment: HPA exists with WVA annotations; pods scale up under load.""" [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] kserve_client = _new_kserve_client() [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] ns = test_case.namespace [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > _create_and_wait(kserve_client, test_case) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:542: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-hpa'], prom... {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] def _create_and_wait(kserve_client, test_case): [e2e-llm-inference-service] """Create LLMISVC and wait for it to be ready.""" [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:482: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...e-hpa-4c186bcf'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-07-29T10:53:27.384801', start_time = 1785322407.3852696 [e2e-llm-inference-service] duration = 900.6076760292053, timestamp_end = '2026-07-29T11:08:27.992948' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....toscale-hpa-4c186bcf'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1376: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7fecdca100e0> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1387: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-29T10:53:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-29T10:53:39Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-29T10:54:06Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-29T10:53:32Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-29T10:54:06Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-29T10:54:06Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-29T10:54:06Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-29T10:54:06Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T10:54:06Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1371: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:82 Created test namespace e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret seaweedfs-s3-creds from kserve-ci-e2e-test to e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret storage-config from kserve-ci-e2e-test to e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:178 Patched default SA in e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 with secret seaweedfs-s3-creds [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:162 ConfigMap odh-kserve-custom-ca-bundle already exists in e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:159 Copied ConfigMap odh-trusted-ca-bundle from kserve-ci-e2e-test to e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig router-managed-autoscale-hpa-de-ec1dce8b in namespace e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig router-managed-autoscale-hpa-de-ec1dce8b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig router-managed-autoscale-hpa-de-ec1dce8b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig workload-llmd-simulator-no-repl-38916baa in namespace e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig workload-llmd-simulator-no-repl-38916baa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig workload-llmd-simulator-no-repl-38916baa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig prometheus-scrape-autoscale-hpa-4c186bcf in namespace e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig prometheus-scrape-autoscale-hpa-4c186bcf [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig prometheus-scrape-autoscale-hpa-4c186bcf [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig scaling-hpa-autoscale-hpa-deplo-347a3180 in namespace e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig scaling-hpa-autoscale-hpa-deplo-347a3180 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig scaling-hpa-autoscale-hpa-deplo-347a3180 [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_autoscaling_hpa_deployment] [2026-07-29T10:53:27.240656] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-hpa'], prompt='KServe is a', service_name='autoscale-hpa-deploy', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, namespace='e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4', before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-deploy', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-de-ec1dce8b'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-38916baa'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-4c186bcf'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-29T10:53:27.257586] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-deploy', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-de-ec1dce8b'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-38916baa'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-4c186bcf'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-29T10:53:27.384615] end - ✅ in 0.127s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-29T10:53:27.384801] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-deploy', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-de-ec1dce8b'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-38916baa'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-4c186bcf'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-29T10:53:32Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-29T10:53:32Z', 'message': 'Inference Pool e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-29T10:53:32Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-29T10:53:32Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-29T10:53:32Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-29T10:53:32Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-29T10:53:32Z', 'message': 'HPA conditions not yet available', 'reason': 'HPAProgressing', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-29T10:53:32Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T10:53:32Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-29T10:53:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-29T10:53:39Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-29T10:53:39Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-29T10:53:32Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-29T10:53:39Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-29T10:53:39Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-29T10:53:32Z', 'message': 'HPA conditions not yet available', 'reason': 'HPAProgressing', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-29T10:53:39Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T10:53:39Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-29T10:53:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-29T10:53:39Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-29T10:54:06Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-29T10:53:32Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-29T10:54:06Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-29T10:54:06Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-29T10:54:06Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-29T10:54:06Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T10:54:06Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1391 Timed out waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-29T10:53:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-29T10:53:39Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-29T10:54:06Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-29T10:53:32Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-29T10:54:06Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-29T10:54:06Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-29T10:54:06Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-29T10:54:06Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T10:54:06Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-07-29T11:08:27.992948] end - ❌ 900.608s: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-29T10:53:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-29T10:53:39Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-29T10:54:06Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-29T10:53:32Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-29T10:54:06Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-29T10:54:06Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-29T10:54:06Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-29T10:54:06Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T10:54:06Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [delete_llmisvc] [2026-07-29T11:08:27.993089] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-deploy', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-de-ec1dce8b'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-38916baa'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-4c186bcf'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 2 pod(s) for autoscale-hpa-deploy still terminating: ['autoscale-hpa-deploy-kserve-66746bd8b9-k8m4k', 'autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-nmc7n'] [e2e-llm-inference-service] assert not ['autoscale-hpa-deploy-kserve-66746bd8b9-k8m4k', 'autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-nmc7n'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 1 pod(s) for autoscale-hpa-deploy still terminating: ['autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-nmc7n'] [e2e-llm-inference-service] assert not ['autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-nmc7n'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [delete_llmisvc] [2026-07-29T11:08:48.097916] end - ✅ in 20.104s [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_autoscaling_hpa_deployment] [2026-07-29T11:08:48.098180] end - ❌ 920.857s: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-29T10:53:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-29T10:53:39Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-29T10:54:06Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-29T10:53:32Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-29T10:54:06Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-29T10:54:06Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-29T10:54:06Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-29T10:54:06Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T10:54:06Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ---------------------------- Captured log teardown ----------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.conftest:conftest.py:168 Skipping deletion of namespace e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 (SKIP_DELETION_ON_FAILURE) [e2e-llm-inference-service] _ test_llm_autoscaling_keda_deployment[router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-keda] _ [e2e-llm-inference-service] [gw1] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-keda'], pro... {'name': 'scaling-keda-autoscale-keda-dep-1ac84077'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.autoscaling_keda [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator-no-replicas", [e2e-llm-inference-service] "prometheus-scrape", [e2e-llm-inference-service] "scaling-keda", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="autoscale-keda-deploy", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_autoscaling_keda_deployment(test_case: TestCase): [e2e-llm-inference-service] """KEDA + Deployment: ScaledObject exists with WVA annotations; no HPA; pods scale up under load.""" [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] kserve_client = _new_kserve_client() [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] ns = test_case.namespace [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > _create_and_wait(kserve_client, test_case) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:606: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-keda'], pro... {'name': 'scaling-keda-autoscale-keda-dep-1ac84077'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] def _create_and_wait(kserve_client, test_case): [e2e-llm-inference-service] """Create LLMISVC and wait for it to be ready.""" [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:482: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...e-ked-101f2a9d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-dep-1ac84077'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-07-29T11:08:48.802533', start_time = 1785323328.8028238 [e2e-llm-inference-service] duration = 900.7736568450928, timestamp_end = '2026-07-29T11:23:49.576505' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....toscale-ked-101f2a9d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-dep-1ac84077'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1376: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7fecdc865620> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1387: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-29T11:09:08Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-29T11:09:08Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-29T11:09:10Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:09:01Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-29T11:09:36Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-29T11:09:36Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-29T11:09:08Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-29T11:09:36Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:09:10Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1371: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:82 Created test namespace e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret seaweedfs-s3-creds from kserve-ci-e2e-test to e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret storage-config from kserve-ci-e2e-test to e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:178 Patched default SA in e2e-test-llm-autoscaling-keda-deployment-b2150d0b with secret seaweedfs-s3-creds [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:162 ConfigMap odh-kserve-custom-ca-bundle already exists in e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:159 Copied ConfigMap odh-trusted-ca-bundle from kserve-ci-e2e-test to e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig router-managed-autoscale-keda-d-27e06c40 in namespace e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig router-managed-autoscale-keda-d-27e06c40 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig router-managed-autoscale-keda-d-27e06c40 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig workload-llmd-simulator-no-repl-da49e827 in namespace e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig workload-llmd-simulator-no-repl-da49e827 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig workload-llmd-simulator-no-repl-da49e827 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig prometheus-scrape-autoscale-ked-101f2a9d in namespace e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig prometheus-scrape-autoscale-ked-101f2a9d [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig prometheus-scrape-autoscale-ked-101f2a9d [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig scaling-keda-autoscale-keda-dep-1ac84077 in namespace e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig scaling-keda-autoscale-keda-dep-1ac84077 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig scaling-keda-autoscale-keda-dep-1ac84077 [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_autoscaling_keda_deployment] [2026-07-29T11:08:48.651454] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-keda'], prompt='KServe is a', service_name='autoscale-keda-deploy', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, namespace='e2e-test-llm-autoscaling-keda-deployment-b2150d0b', before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-keda-deploy', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-keda-deployment-b2150d0b', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-keda-d-27e06c40'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-da49e827'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-ked-101f2a9d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-dep-1ac84077'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-29T11:08:48.663857] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-keda-deploy', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-keda-deployment-b2150d0b', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-keda-d-27e06c40'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-da49e827'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-ked-101f2a9d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-dep-1ac84077'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-29T11:08:48.802376] end - ✅ in 0.138s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-29T11:08:48.802533] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-keda-deploy', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-keda-deployment-b2150d0b', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-keda-d-27e06c40'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-da49e827'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-ked-101f2a9d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-dep-1ac84077'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-29T11:09:01Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-29T11:09:01Z', 'message': 'Inference Pool e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-29T11:09:01Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:09:01Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-29T11:09:01Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-29T11:09:01Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-29T11:09:01Z', 'message': 'ScaledObject not yet visible in cache', 'reason': 'ScaledObjectProgressing', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-29T11:09:01Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:09:01Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-29T11:09:08Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-route: "False" (reason "BackendNotFound", message "backend(autoscale-keda-deploy-inference-pool-ip-d28561d2.e2e-test-llm-autoscaling-keda-deployment-b2150d0b.svc.cluster.local) not found")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-29T11:09:08Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-29T11:09:08Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:09:08Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-29T11:09:08Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-route: "False" (reason "BackendNotFound", message "backend(autoscale-keda-deploy-inference-pool-ip-d28561d2.e2e-test-llm-autoscaling-keda-deployment-b2150d0b.svc.cluster.local) not found")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-29T11:09:08Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-route: "False" (reason "BackendNotFound", message "backend(autoscale-keda-deploy-inference-pool-ip-d28561d2.e2e-test-llm-autoscaling-keda-deployment-b2150d0b.svc.cluster.local) not found")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-29T11:09:08Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-29T11:09:08Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:09:08Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-29T11:09:08Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-29T11:09:08Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-29T11:09:08Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:09:01Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-29T11:09:08Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-29T11:09:08Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-29T11:09:08Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-29T11:09:08Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:09:08Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-29T11:09:08Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-29T11:09:08Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-29T11:09:10Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:09:01Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-29T11:09:08Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-29T11:09:08Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-29T11:09:08Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-29T11:09:08Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:09:10Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-29T11:09:08Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-29T11:09:08Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-29T11:09:10Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:09:01Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-29T11:09:36Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-29T11:09:36Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-29T11:09:08Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-29T11:09:36Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:09:10Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1391 Timed out waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-29T11:09:08Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-29T11:09:08Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-29T11:09:10Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:09:01Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-29T11:09:36Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-29T11:09:36Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-29T11:09:08Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-29T11:09:36Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:09:10Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-07-29T11:23:49.576505] end - ❌ 900.774s: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-29T11:09:08Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-29T11:09:08Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-29T11:09:10Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:09:01Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-29T11:09:36Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-29T11:09:36Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-29T11:09:08Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-29T11:09:36Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:09:10Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [delete_llmisvc] [2026-07-29T11:23:49.576908] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-keda-deploy', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-keda-deployment-b2150d0b', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-keda-d-27e06c40'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-da49e827'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-ked-101f2a9d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-dep-1ac84077'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 2 pod(s) for autoscale-keda-deploy still terminating: ['autoscale-keda-deploy-kserve-77fc6d7cf9-82vxl', 'autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-hw94r'] [e2e-llm-inference-service] assert not ['autoscale-keda-deploy-kserve-77fc6d7cf9-82vxl', 'autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-hw94r'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 1 pod(s) for autoscale-keda-deploy still terminating: ['autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-hw94r'] [e2e-llm-inference-service] assert not ['autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-hw94r'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [delete_llmisvc] [2026-07-29T11:24:19.880665] end - ✅ in 30.303s [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_autoscaling_keda_deployment] [2026-07-29T11:24:19.880806] end - ❌ 931.229s: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-29T11:09:08Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-29T11:09:08Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-29T11:09:10Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:09:01Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-29T11:09:36Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-29T11:09:36Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-29T11:09:08Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-29T11:09:36Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:09:10Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ---------------------------- Captured log teardown ----------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.conftest:conftest.py:168 Skipping deletion of namespace e2e-test-llm-autoscaling-keda-deployment-b2150d0b (SKIP_DELETION_ON_FAILURE) [e2e-llm-inference-service] _____ test_llm_inference_service[router-managed-workload-pd-cpu-model-pvc] _____ [e2e-llm-inference-service] [gw0] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-pd-cpu', 'model-pvc'], prompt='KServe is a', service_name='llmisvc-mod... {'name': 'model-pvc-llmisvc-model-pvc-rou-49c1f027'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-gateway-ref", [e2e-llm-inference-service] "router-with-managed-route", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] expected_gateway="router-gateway-1", [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda tc: create_router_resources( [e2e-llm-inference-service] gateways=[ [e2e-llm-inference-service] make_router_gateway( [e2e-llm-inference-service] "router-gateway-1", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="custom-route-timeout-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="router-with-refs-test", [e2e-llm-inference-service] expected_gateway="router-gateway-1", [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda tc: create_router_resources( [e2e-llm-inference-service] gateways=[ [e2e-llm-inference-service] make_router_gateway( [e2e-llm-inference-service] "router-gateway-1", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] routes=[ [e2e-llm-inference-service] make_router_main_route( [e2e-llm-inference-service] "router-route-1", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] "router-gateway-1", [e2e-llm-inference-service] "router-with-refs-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] make_router_health_route( [e2e-llm-inference-service] "router-route-2", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] "router-gateway-1", [e2e-llm-inference-service] "router-with-refs-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=["router-managed", "workload-pd-cpu", "model-fb-opt-125m"], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="custom-route-timeout-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="router-with-refs-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] expected_gateway="router-gateway-2", [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda tc: create_router_resources( [e2e-llm-inference-service] gateways=[ [e2e-llm-inference-service] make_router_gateway( [e2e-llm-inference-service] "router-gateway-2", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] routes=[ [e2e-llm-inference-service] make_router_main_route( [e2e-llm-inference-service] "router-route-3", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] "router-gateway-2", [e2e-llm-inference-service] "router-with-refs-pd-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] make_router_health_route( [e2e-llm-inference-service] "router-route-4", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] "router-gateway-2", [e2e-llm-inference-service] "router-with-refs-pd-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-dp-ep-gpu", [e2e-llm-inference-service] "workload-dp-ep-prefill-gpu", [e2e-llm-inference-service] "model-deepseek-v2-lite", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="Delve into the multifaceted implications of a fully disaggregated cloud architecture, specifically " [e2e-llm-inference-service] "where the compute plane (P) and the data plane (D) are independently deployed and managed for a " [e2e-llm-inference-service] "geographically distributed, high-throughput, low-latency microservices ecosystem. Beyond the " [e2e-llm-inference-service] "fundamental challenges of network latency and data consistency, elaborate on the advanced " [e2e-llm-inference-service] "considerations and trade-offs inherent in such a setup: 1. Network Architecture and Protocols: " [e2e-llm-inference-service] "How would the network fabric and underlying protocols (e.g., RDMA, custom transport layers) need to " [e2e-llm-inference-service] "evolve to support optimal performance and minimize inter-plane communication overhead, especially for " [e2e-llm-inference-service] "synchronous operations? Discuss the role of network programmability (e.g., SDN, P4) in dynamically " [e2e-llm-inference-service] "optimizing routing and traffic flow between P and D. 2. Advanced Data Consistency and Durability: " [e2e-llm-inference-service] "Explore sophisticated data consistency models (e.g., causal consistency, strong eventual consistency) " [e2e-llm-inference-service] "and their applicability in balancing performance and data integrity across a globally distributed data plane. " [e2e-llm-inference-service] "Detail strategies for ensuring data durability and fault tolerance, including multi-region replication, " [e2e-llm-inference-service] "intelligent partitioning, and recovery mechanisms in the event of partial or full plane failures. " [e2e-llm-inference-service] "3. Dynamic Resource Orchestration and Cost Optimization: Analyze how an orchestration layer would intelligently " [e2e-llm-inference-service] "manage the independent scaling of compute (P) and data (D) resources, considering fluctuating workloads, " [e2e-llm-inference-service] "cost efficiency, and performance targets (e.g., using predictive analytics for resource provisioning). " [e2e-llm-inference-service] "Discuss mechanisms for dynamically reallocating compute nodes to different data partitions based on " [e2e-llm-inference-service] "workload patterns and data locality, potentially involving live migration strategies. " [e2e-llm-inference-service] "4. Security and Compliance in a Distributed Landscape: Address the enhanced security perimeter " [e2e-llm-inference-service] "challenges, including securing communication channels between P and D (encryption in transit, mutual TLS), " [e2e-llm-inference-service] "fine-grained access control to data at rest and in motion, and identity management across disaggregated " [e2e-llm-inference-service] "components. Discuss how such an architecture impacts compliance with regulatory frameworks (e.g., GDPR, HIPAA) " [e2e-llm-inference-service] "concerning data sovereignty, privacy, and auditability. 5. Operational Complexity and Observability: " [e2e-llm-inference-service] "Examine the increased complexity in monitoring, logging, and tracing across highly decoupled compute and " [e2e-llm-inference-service] "data planes. What specialized tooling and practices (e.g., distributed tracing with OpenTelemetry, advanced AIOps) " [e2e-llm-inference-service] "would be essential? How would incident response and troubleshooting differ in this disaggregated environment " [e2e-llm-inference-service] "compared to traditional integrated systems? Consider the challenges of pinpointing root causes across " [e2e-llm-inference-service] "independent failures. 6. Real-world Applicability and Future Trends: Identify specific industries " [e2e-llm-inference-service] "or use cases (e.g., high-frequency trading, IoT edge processing, large language model inference) " [e2e-llm-inference-service] "where the benefits of P/D disaggregation would strongly outweigh its complexities. " [e2e-llm-inference-service] "Conclude by speculating on emerging technologies or paradigms (e.g., serverless compute functions " [e2e-llm-inference-service] "directly interacting with object storage, in-memory disaggregation) that could further drive or " [e2e-llm-inference-service] "transform P/D disaggregation in cloud computing.", [e2e-llm-inference-service] max_tokens=2000, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_gpu, [e2e-llm-inference-service] pytest.mark.cluster_nvidia, [e2e-llm-inference-service] pytest.mark.cluster_nvidia_roce, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-no-scheduler", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.no_scheduler, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="This test simulates DP+EP that can run on CPU, the idea is to test the LWS-based deployment, " [e2e-llm-inference-service] "but without the resources requirements for DP+EP (GPUs and ROCe/IB).", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_multi_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler config tests [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-inline-config-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Chat completions endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] model_name="Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-configmap-ref", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-configmap-ref-test", [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda tc: create_scheduler_configmap(namespace=tc.namespace) [e2e-llm-inference-service] ], [e2e-llm-inference-service] after_test=[ [e2e-llm-inference-service] lambda tc: delete_scheduler_configmap(namespace=tc.namespace) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-replicas", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-ha-replicas-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-custom-template", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-custom-template-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler v0.6 → v0.7 migration tests. [e2e-llm-inference-service] # Deploy v0.6-style configs and verify the controller migrates them [e2e-llm-inference-service] # so the v0.7 scheduler boots successfully. [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-v06-pd-config-migration", [e2e-llm-inference-service] "workload-llmd-simulator-pd", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-v06-pd-migration-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-v06-nonzero-threshold-migration", [e2e-llm-inference-service] "workload-llmd-simulator-pd", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-v06-threshold-migration-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Standalone tokenizer — clean path: token-producer in inline config [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-tokenizer-kvcache", [e2e-llm-inference-service] "workload-llmd-simulator-kvcache", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="tokenizer-clean-path-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Standalone tokenizer — migration path: legacy precise-prefix-cache-scorer [e2e-llm-inference-service] # triggers auto-provisioned tokenizer without explicit tokenizer:{} field [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-precise-prefix-cache-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator-kvcache", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="tokenizer-migration-path-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Models endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="data"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/chat/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — LoRA adapter [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] model_name="publishers/{namespace}/models/lora-adapter-1", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "publishers/{namespace}/models/lora-adapter-1" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/lora-adapter-1", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/models (base + LoRA) [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=assert_models_contains( [e2e-llm-inference-service] "facebook/opt-125m", [e2e-llm-inference-service] "publishers/{namespace}/models/facebook/opt-125m", [e2e-llm-inference-service] "lora-adapter-1", [e2e-llm-inference-service] "publishers/{namespace}/models/lora-adapter-1", [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # PVC storage tests -- validate direct PVC volume mount with real vLLM serving [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] before_test=[lambda tc: ensure_pvc_with_model(namespace=tc.namespace)], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] before_test=[lambda tc: ensure_pvc_with_model(namespace=tc.namespace)], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] before_test=[lambda tc: ensure_pvc_with_model(namespace=tc.namespace)], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_multi_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_inference_service(test_case: TestCase): # noqa: F811 [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = KServeClient( [e2e-llm-inference-service] config_file=os.environ.get("KUBECONFIG", "~/.kube/config"), [e2e-llm-inference-service] client_configuration=client.Configuration(), [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] prefix = test_case.log_prefix [e2e-llm-inference-service] [e2e-llm-inference-service] test_failed = False [e2e-llm-inference-service] try: [e2e-llm-inference-service] print(f"{prefix} Creating LLMInferenceService {service_name}") [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] print(f"{prefix} Waiting for LLMInferenceService {service_name} to be ready") [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:866: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...del-p-9d807ba3'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-49c1f027'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-07-29T11:12:01.675538', start_time = 1785323521.675835 [e2e-llm-inference-service] duration = 900.6989421844482, timestamp_end = '2026-07-29T11:27:02.374780' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....svc-model-p-9d807ba3'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-49c1f027'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1376: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7f21ca0c28e0> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1387: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-07-29T11:12:35Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-29T11:12:35Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-29T11:14:04Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:12:35Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:12:06Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-29T11:12:35Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-29T11:12:41Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-29T11:12:41Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:12:35Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1371: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:82 Created test namespace e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret seaweedfs-s3-creds from kserve-ci-e2e-test to e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret storage-config from kserve-ci-e2e-test to e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:178 Patched default SA in e2e-test-llm-inference-service-48639af5 with secret seaweedfs-s3-creds [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:162 ConfigMap odh-kserve-custom-ca-bundle already exists in e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:159 Copied ConfigMap odh-trusted-ca-bundle from kserve-ci-e2e-test to e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1959 Created PVC e2e-pvc-model-storage in namespace e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2099 Created model download Job e2e-pvc-model-download in namespace e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 6: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-29T10:41:39Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2124 Model download Job e2e-pvc-model-download completed successfully [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig router-managed-llmisvc-model-pv-d968e7b0 in namespace e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig router-managed-llmisvc-model-pv-d968e7b0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig router-managed-llmisvc-model-pv-d968e7b0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig workload-pd-cpu-llmisvc-model-p-9d807ba3 in namespace e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig workload-pd-cpu-llmisvc-model-p-9d807ba3 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig workload-pd-cpu-llmisvc-model-p-9d807ba3 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig model-pvc-llmisvc-model-pvc-rou-49c1f027 in namespace e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig model-pvc-llmisvc-model-pvc-rou-49c1f027 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig model-pvc-llmisvc-model-pvc-rou-49c1f027 [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_inference_service] [2026-07-29T11:12:01.512068] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-pd-cpu', 'model-pvc'], prompt='KServe is a', service_name='llmisvc-model-pvc-router-manage-e8706282', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, namespace='e2e-test-llm-inference-service-48639af5', before_test=[ at 0x7f21ca71fc40>], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-pvc-router-manage-e8706282', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-inference-service-48639af5', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-pv-d968e7b0'}, [e2e-llm-inference-service] {'name': 'workload-pd-cpu-llmisvc-model-p-9d807ba3'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-49c1f027'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-29T11:12:01.526384] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-pvc-router-manage-e8706282', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-inference-service-48639af5', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-pv-d968e7b0'}, [e2e-llm-inference-service] {'name': 'workload-pd-cpu-llmisvc-model-p-9d807ba3'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-49c1f027'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-29T11:12:01.675375] end - ✅ in 0.149s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-29T11:12:01.675538] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-pvc-router-manage-e8706282', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-inference-service-48639af5', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-pv-d968e7b0'}, [e2e-llm-inference-service] {'name': 'workload-pd-cpu-llmisvc-model-p-9d807ba3'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-49c1f027'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'RouterReady', 'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-07-29T11:12:06Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-29T11:12:06Z', 'message': 'Inference Pool e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-29T11:12:06Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:12:06Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:12:06Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-29T11:12:06Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-29T11:12:06Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-29T11:12:06Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:12:06Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'RouterReady', 'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-07-29T11:12:35Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-29T11:12:35Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-29T11:12:35Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:12:35Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:12:06Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-29T11:12:35Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-29T11:12:35Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-29T11:12:35Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:12:35Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-07-29T11:12:35Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-29T11:12:35Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-29T11:12:35Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:12:35Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:12:06Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-29T11:12:35Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-29T11:12:41Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-29T11:12:41Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:12:35Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-07-29T11:12:35Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-29T11:12:35Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-29T11:14:04Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:12:35Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:12:06Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-29T11:12:35Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-29T11:12:41Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-29T11:12:41Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:12:35Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 7: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-29T10:41:39Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 8: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-29T10:41:39Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 9: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-29T10:41:39Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1391 Timed out waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-07-29T11:12:35Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-29T11:12:35Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-29T11:14:04Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:12:35Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:12:06Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-29T11:12:35Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-29T11:12:41Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-29T11:12:41Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:12:35Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-07-29T11:27:02.374780] end - ❌ 900.699s: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-07-29T11:12:35Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-29T11:12:35Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-29T11:14:04Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:12:35Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:12:06Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-29T11:12:35Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-29T11:12:41Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-29T11:12:41Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:12:35Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:903 [router-managed-workload-pd-cpu-model-pvc] ❌ ERROR: Failed to call llm inference service llmisvc-model-pvc-router-manage-e8706282: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-07-29T11:12:35Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-29T11:12:35Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-29T11:14:04Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:12:35Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:12:06Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-29T11:12:35Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-29T11:12:41Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-29T11:12:41Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:12:35Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:243 # Diagnostics for 'llmisvc-model-pvc-router-manage-e8706282' in 'e2e-test-llm-inference-service-48639af5' [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:244 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:247 # LLMInferenceService llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:256 apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] security.opendatahub.io/enable-auth: 'false' [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:12:01Z' [e2e-llm-inference-service] finalizers: [e2e-llm-inference-service] - serving.kserve.io/llmisvc-finalizer [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:security.opendatahub.io/enable-auth: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:baseRefs: {} [e2e-llm-inference-service] manager: OpenAPI-Generator [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-29T11:12:01Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:finalizers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] v:"serving.kserve.io/llmisvc-finalizer": {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-29T11:12:01Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:addresses: {} [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-router-route: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-scheduler: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-scheduler-latency-predictor: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-tokenizer: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-tracing: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-worker-data-parallel: {} [e2e-llm-inference-service] f:appliedConfigs: {} [e2e-llm-inference-service] f:conditions: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:router: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:gateways: {} [e2e-llm-inference-service] f:scheduler: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:inferencePool: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:service: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:url: {} [e2e-llm-inference-service] f:workloads: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:prefill: {} [e2e-llm-inference-service] f:primary: {} [e2e-llm-inference-service] f:scheduler: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-29T11:14:04Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] resourceVersion: '57670' [e2e-llm-inference-service] uid: 7031a75f-c7bb-4cfb-921d-6fc95e619eb3 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] baseRefs: [e2e-llm-inference-service] - name: router-managed-llmisvc-model-pv-d968e7b0 [e2e-llm-inference-service] - name: workload-pd-cpu-llmisvc-model-p-9d807ba3 [e2e-llm-inference-service] - name: model-pvc-llmisvc-model-pvc-rou-49c1f027 [e2e-llm-inference-service] model: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uri: '' [e2e-llm-inference-service] status: [e2e-llm-inference-service] addresses: [e2e-llm-inference-service] - name: gateway-external-model-routing [e2e-llm-inference-service] url: http://a4d5e2ea28ab3498a9ca1aabff76d988-1031208529.us-east-1.elb.amazonaws.com/ [e2e-llm-inference-service] - name: gateway-external [e2e-llm-inference-service] url: http://a4d5e2ea28ab3498a9ca1aabff76d988-1031208529.us-east-1.elb.amazonaws.com/e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] - name: gateway-external [e2e-llm-inference-service] url: http://a4d5e2ea28ab3498a9ca1aabff76d988-1031208529.us-east-1.elb.amazonaws.com/publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] - name: gateway-internal-model-routing [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/ [e2e-llm-inference-service] - name: gateway-internal [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] - name: gateway-internal [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-template: kserve-config-llm-decode-template [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-worker-data-parallel: kserve-config-llm-decode-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-template: kserve-config-llm-prefill-template [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-worker-data-parallel: kserve-config-llm-prefill-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-router-route: kserve-config-llm-router-route [e2e-llm-inference-service] serving.kserve.io/config-llm-scheduler: kserve-config-llm-scheduler [e2e-llm-inference-service] serving.kserve.io/config-llm-scheduler-latency-predictor: kserve-config-llm-scheduler-latency-predictor [e2e-llm-inference-service] serving.kserve.io/config-llm-template: kserve-config-llm-template [e2e-llm-inference-service] serving.kserve.io/config-llm-tokenizer: kserve-config-llm-tokenizer [e2e-llm-inference-service] serving.kserve.io/config-llm-tracing: kserve-config-llm-tracing [e2e-llm-inference-service] serving.kserve.io/config-llm-worker-data-parallel: kserve-config-llm-worker-data-parallel [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-29T11:12:35Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: HTTPRoutesReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-29T11:12:35Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: InferencePoolReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-29T11:14:04Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: MainWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-29T11:12:35Z' [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: PrefillWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-29T11:12:06Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: PresetsCombined [e2e-llm-inference-service] - lastTransitionTime: '2026-07-29T11:12:35Z' [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: Ready [e2e-llm-inference-service] - lastTransitionTime: '2026-07-29T11:12:41Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: RouterReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-29T11:12:41Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: SchedulerWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-29T11:12:35Z' [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: WorkloadsReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] url: http://a4d5e2ea28ab3498a9ca1aabff76d988-1031208529.us-east-1.elb.amazonaws.com/e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:68 TIME NAMESPACE SOURCE TYPE REASON MESSAGE [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:69 -------------------------------------------------------------------------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None e2e-test-llm-inference-service-48639af5 Normal Scheduled Successfully assigned e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-hcdhx to ip-10-0-133-86.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:11:48 e2e-test-llm-inference-service-48639af5 attachdetach-controller Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-ce3748df-ddb8-4bce-a255-9229c58859db" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:11:48 e2e-test-llm-inference-service-48639af5 multus Normal AddedInterface Add eth0 [10.132.0.60/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:11:48 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-133-86.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:13da3ee375cce7c9d4590e64ebb24ecf2e97ed4b2b64f9a4f2e3074ed4aafac1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:11:48 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-133-86.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:11:48 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-133-86.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:11:41 e2e-test-llm-inference-service-48639af5 job-controller Normal SuccessfulCreate Created pod: e2e-pvc-model-download-hcdhx [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:11:57 e2e-test-llm-inference-service-48639af5 job-controller Normal Completed Job completed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:11:41 e2e-test-llm-inference-service-48639af5 persistentvolume-controller Normal WaitForFirstConsumer waiting for first consumer to be created before binding [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:11:41 e2e-test-llm-inference-service-48639af5 ebs.csi.aws.com_aws-ebs-csi-driver-controller-846c7568bf-vmfhs_43046ce3-27b6-434f-8292-da6b37871cde Normal Provisioning External provisioner is provisioning volume for claim "e2e-test-llm-inference-service-48639af5/e2e-pvc-model-storage" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:11:41 e2e-test-llm-inference-service-48639af5 persistentvolume-controller Normal ExternalProvisioning Waiting for a volume to be created either by the external provisioner 'ebs.csi.aws.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:11:43 e2e-test-llm-inference-service-48639af5 ebs.csi.aws.com_aws-ebs-csi-driver-controller-846c7568bf-vmfhs_43046ce3-27b6-434f-8292-da6b37871cde Normal ProvisioningSucceeded Successfully provisioned volume pvc-ce3748df-ddb8-4bce-a255-9229c58859db [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:12:03 e2e-test-llm-inference-service-48639af5 replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None e2e-test-llm-inference-service-48639af5 Normal Scheduled Successfully assigned e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67 to ip-10-0-133-86.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:12:07 e2e-test-llm-inference-service-48639af5 multus Normal AddedInterface Add eth0 [10.132.0.61/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:12:07 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-133-86.ec2.internal Normal Pulled Container image "quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.9.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:12:07 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-133-86.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:12:07 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-133-86.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:12:08 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-133-86.ec2.internal Normal Pulled Container image "vllm/vllm-openai-cpu:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:12:08 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-133-86.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:12:08 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-133-86.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:13:47 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-133-86.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.61:8001/health": dial tcp 10.132.0.61:8001: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:12:03 e2e-test-llm-inference-service-48639af5 replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None e2e-test-llm-inference-service-48639af5 Normal Scheduled Successfully assigned e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk to ip-10-0-129-80.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:12:03 e2e-test-llm-inference-service-48639af5 attachdetach-controller Warning FailedAttachVolume Multi-Attach error for volume "pvc-ce3748df-ddb8-4bce-a255-9229c58859db" Volume is already used by pod(s) llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:12:03 e2e-test-llm-inference-service-48639af5 deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b89cd7957 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:12:03 e2e-test-llm-inference-service-48639af5 deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:12:04 e2e-test-llm-inference-service-48639af5 OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-pvc-router-manage-e8706282-kserve-route-authn: failed to get HTTPRoute e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-pvc-router-manage-e8706282-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:12:03 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.Secret e2e-test-llm-inference-service-48639af5/llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:12:03 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.ServiceAccount e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:12:03 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.Role e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:12:03 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.RoleBinding e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:12:03 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.Deployment e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:12:03 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.Deployment e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:12:03 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.Service e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:12:03 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.ServiceAccount e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:12:03 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /e2e-test-llm-inference-service-132820f03254470cd8a0146008974a5f [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:12:34 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:22:04 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Updated Updated v1.Secret e2e-test-llm-inference-service-48639af5/llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:12:34 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:12:04 e2e-test-llm-inference-service-48639af5 replicaset-controller Normal SuccessfulCreate Created pod: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schexzxnd [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:12:04 e2e-test-llm-inference-service-48639af5 deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-7fc69df7b8 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None e2e-test-llm-inference-service-48639af5 Normal Scheduled Successfully assigned e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schexzxnd to ip-10-0-140-240.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:12:04 e2e-test-llm-inference-service-48639af5 multus Normal AddedInterface Add eth0 [10.134.0.45/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:12:04 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-140-240.ec2.internal Normal Pulled Container image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:12:04 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-140-240.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:12:04 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-140-240.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:172 ### Pod llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67 (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:212 #### init-container 'llm-d-routing-sidecar' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:225 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:226 {"level":"info","ts":1785323527.9944203,"msg":"Proxy starting","Built on":"","From Git SHA":"unknown"} [e2e-llm-inference-service] {"level":"info","ts":1785323527.9944642,"msg":"Proxy configuration","config":"{\"Port\":\"8000\",\"KVConnector\":\"nixlv2\",\"ECConnector\":\"\",\"DataParallelSize\":1,\"MaxIdleConnsPerHost\":1024,\"EnablePrefillerSampling\":false,\"PrefillMaxRetries\":0,\"PrefillRetryBackoff\":200000000,\"UseTLSForPrefiller\":true,\"UseTLSForDecoder\":true,\"UseTLSForEncoder\":false,\"InsecureSkipVerifyForPrefiller\":false,\"InsecureSkipVerifyForEncoder\":false,\"InsecureSkipVerifyForDecoder\":false,\"SecureServing\":true,\"CertPath\":\"/var/run/kserve/tls\",\"MooncakeBootstrapPort\":8998,\"EnableSSRFProtection\":true,\"InferencePoolNamespace\":\"e2e-test-llm-inference-service-48639af5\",\"InferencePoolName\":\"llmisvc-model-pvc-router-manage-e8706282-inference-pool\",\"PoolGroup\":\"inference.networking.x-k8s.io\",\"DecodeChunkSize\":0,\"Tracing\":false,\"MoRIIOWriteMode\":false,\"MoRIIODecodeNotifyPort\":61005,\"MoRIIODecodeHandshakePort\":6301,\"MoRIIODecodePodIP\":\"\",\"MoRIIOParallelDispatch\":false,\"MoRIIOPrefillHandshakePort\":6301,\"MoRIIOPrefillNotifyPort\":61005,\"MoRIIOTPSize\":1,\"MoRIIODPSize\":1,\"MoRIIORemoteHosts\":null,\"MoRIIODPSizeLocal\":0,\"MoRIIODecodeHosts\":null,\"DecoderURL\":\"https://localhost:8001\"}"} [e2e-llm-inference-service] {"level":"info","ts":1785323527.9950724,"logger":"allowlist-validator","msg":"starting SSRF protection allowlist validator","namespace":"e2e-test-llm-inference-service-48639af5","poolName":"llmisvc-model-pvc-router-manage-e8706282-inference-pool","gvr":"inference.networking.x-k8s.io/v1alpha2, Resource=inferencepools"} [e2e-llm-inference-service] {"level":"info","ts":1785323528.00767,"logger":"allowlist-validator","msg":"InferencePool added","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785323528.0152023,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":0,"targets":{}} [e2e-llm-inference-service] {"level":"info","ts":1785323528.0152757,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":0,"targets":{}} [e2e-llm-inference-service] {"level":"info","ts":1785323528.0955224,"logger":"allowlist-validator","msg":"allowlist validator started successfully"} [e2e-llm-inference-service] {"level":"info","ts":1785323528.0972517,"logger":"proxy server on port 8000","msg":"server TLS configured"} [e2e-llm-inference-service] {"level":"info","ts":1785323528.0972705,"logger":"proxy server on port 8000","msg":"starting","addr":"[::]:8000"} [e2e-llm-inference-service] {"level":"info","ts":1785323528.6712391,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.61":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785323529.6781383,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.61":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785323537.8189816,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.61":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785323558.0086324,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785323558.0195444,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.61":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785323558.0195963,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.61":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785323588.008964,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785323588.0181487,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.61":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785323588.0182428,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.61":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785323618.0098307,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785323618.019456,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.61":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785323618.0195103,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.61":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785323637.8186705,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.61":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785323637.839413,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.61":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785323648.0101724,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785323648.0195265,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.61":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785323648.019577,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.61":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785323678.010474,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785323678.020526,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.61":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785323678.0205843,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.61":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785323708.0107946,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785323708.0199213,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.61":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785323708.0199745,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.61":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785323738.0113952,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785323738.020555,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.61":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785323738.0206072,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.61":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785323768.0118477,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785323768.0210073,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.61":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785323768.0210562,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.61":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785323798.0127692,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785323798.0223904,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.61":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785323798.0226738,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.61":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785323828.0129664,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785323828.0219355,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.61":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785323828.0219846,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.61":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785323858.0138824,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785323858.0239635,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.61":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785323858.0240192,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.61":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785323888.014097,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785323888.0230327,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.61":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785323888.0230813,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.61":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785323918.0143747,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785323918.0242774,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.61":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785323918.024333,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.61":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785323948.0145965,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785323948.0242448,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.61":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785323948.0243,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.61":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785323978.0150135,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785323978.0249856,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.61":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785323978.025038,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.61":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785324008.0159338,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785324008.026181,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.61":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785324008.0262427,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.61":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785324038.0169587,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785324038.0269988,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.61":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785324038.0270474,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.61":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785324068.017971,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785324068.0278058,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.61":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785324068.0278618,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.61":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785324098.0180488,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785324098.0279956,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.61":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785324098.0280454,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.61":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785324128.018784,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785324128.0280933,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.61":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785324128.0281634,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.61":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785324158.019179,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785324158.0292473,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.61":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785324158.0293007,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.61":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785324188.0199904,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785324188.0293136,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.61":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785324188.029361,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.61":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785324218.0207286,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785324218.0302744,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.61":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785324218.0303228,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.61":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785324248.0207658,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785324248.0299942,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.61":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785324248.0300455,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.61":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785324278.0215538,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785324278.0307448,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.61":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785324278.030789,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.61":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785324308.0217447,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785324308.0312,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.61":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785324308.0312662,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.61":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785324338.0218294,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785324338.0309546,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.61":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785324338.03115,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.61":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785324368.0228486,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785324368.033021,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.61":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785324368.033077,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.61":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785324398.0236752,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785324398.033289,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.61":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785324398.0333447,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.61":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67":{}}} [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:212 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:225 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:226 [W729 11:13:16.593776432 utils.cpp:76] Warning: numa_migrate_pages failed. errno: 1 (function init_cpu_threads_env) [e2e-llm-inference-service] [W729 11:13:16.593802103 utils.cpp:103] Warning: NUMA binding: Using MEMBIND policy for memory allocation on the NUMA nodes (0). Memory allocations will be strictly bound to these NUMA nodes. (function init_cpu_threads_env) [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-29 11:13:16 [v1/worker/cpu_worker.py:109] OMP threads binding of Process 74: [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-29 11:13:16 [v1/worker/cpu_worker.py:109] OMP tid: 74, core 4 [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-29 11:13:16 [v1/worker/cpu_worker.py:109] OMP tid: 91, core 5 [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-29 11:13:16 [v1/worker/cpu_worker.py:109] OMP tid: 92, core 6 [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-29 11:13:16 [v1/worker/cpu_worker.py:109] OMP tid: 93, core 7 [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-29 11:13:16 [v1/worker/cpu_worker.py:109] [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-29 11:13:16 [distributed/parallel_state.py:1356] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.132.0.61:34693 backend=gloo [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-29 11:13:16 [distributed/parallel_state.py:1400] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.132.0.61:34693 backend=gloo [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-29 11:13:16 [distributed/parallel_state.py:1459] Detected 1 nodes in the distributed environment [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-29 11:13:16 [distributed/parallel_state.py:1716] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-29 11:13:16 [v1/sample/logits_processor/__init__.py:65] No logitsprocs plugins installed (group vllm.logits_processors). [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-29 11:13:16 [model_executor/offloader/base.py:107] Offloader set to NoopOffloader (no offloading). [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-29 11:13:16 [v1/worker/cpu_model_runner.py:71] Starting to load model /mnt/models... [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-29 11:13:16 [compilation/decorators.py:213] Inferred dynamic dimensions for forward method of : ['input_ids', 'positions', 'intermediate_tensors', 'inputs_embeds'] [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-29 11:13:16 [config/compilation.py:1194] enabled custom ops: Counter() [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-29 11:13:16 [config/compilation.py:1195] disabled custom ops: Counter({'vocab_parallel_embedding': 1, 'logits_processor': 1}) [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-29 11:13:16 [model_executor/model_loader/base_loader.py:63] Loading weights on cpu ... [e2e-llm-inference-service] (EngineCore pid=74) Loading pt checkpoint shards: 0% Completed | 0/1 [00:00 [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:13:25 [v1/engine/utils.py:1047] Waiting for 1 local, 0 remote core engine proc(s) to start. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:13:35 [v1/engine/utils.py:1047] Waiting for 1 local, 0 remote core engine proc(s) to start. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:13:45 [v1/engine/utils.py:1047] Waiting for 1 local, 0 remote core engine proc(s) to start. [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-29 11:13:52 [compilation/decorators.py:640] saved AOT compiled function to /home/.cache/vllm/torch_compile_cache/torch_aot_compile/86c9c3c579382eef68a98ac1d59b39811ba08abef3b4e90675a32c8dec3d7c90/rank_0_0/model [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-29 11:13:54 [compilation/monitor.py:76] Initial profiling/warmup run took 1.16 s [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-29 11:13:54 [v1/worker/cpu_model_runner.py:92] Warming up done. [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-29 11:13:54 [v1/engine/core.py:283] init engine (profile, create kv cache, warmup model) took 36.45 seconds [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-29 11:13:54 [tokenizers/registry.py:68] Loading CachedHfTokenizer for tokenizer_mode='hf' [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-29 11:13:54 [utils/gc_utils.py:40] GC Debug Config. enabled:False,top_objects:-1 [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-29 11:13:54 [config/vllm.py:790] Asynchronous scheduling is disabled. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:13:54 [v1/engine/utils.py:1158] READY from local core engine process 0. [e2e-llm-inference-service] (EngineCore pid=74) WARNING 07-29 11:13:54 [config/vllm.py:859] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-29 11:13:54 [v1/engine/core.py:1158] EngineCore waiting for work. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:13:54 [v1/metrics/loggers.py:273] Engine 000: vllm cache_config_info with initialization after num_gpu_blocks is: 227 [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-29 11:13:54 [v1/engine/core.py:1158] EngineCore waiting for work. [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-29 11:13:54 [v1/engine/core.py:1158] EngineCore waiting for work. [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 11:13:54 [entrypoints/openai/api_server.py:590] Supported tasks: ['generate'] [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:13:55 [renderers/base.py:197] Warming up chat template processing... [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:13:55 [renderers/hf.py:122] Failed to load AutoTokenizer chat template for /mnt/models [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:13:55 [renderers/hf.py:122] Traceback (most recent call last): [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:13:55 [renderers/hf.py:122] File "/opt/venv/lib/python3.12/site-packages/vllm/renderers/hf.py", line 120, in resolve_chat_template [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:13:55 [renderers/hf.py:122] return tokenizer.get_chat_template(chat_template, tools=tools) [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:13:55 [renderers/hf.py:122] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:13:55 [renderers/hf.py:122] File "/opt/venv/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 1825, in get_chat_template [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:13:55 [renderers/hf.py:122] raise ValueError( [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:13:55 [renderers/hf.py:122] ValueError: Cannot use chat template functions because tokenizer.chat_template is not set and no template argument was passed! For information about writing templates and setting the tokenizer.chat_template attribute, please see the documentation at https://huggingface.co/docs/transformers/main/en/chat_templating [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:13:55 [renderers/hf.py:141] There is no chat template fallback for /mnt/models [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 11:13:55 [renderers/hf.py:314] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:13:55 [renderers/hf.py:122] Failed to load AutoTokenizer chat template for /mnt/models [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:13:55 [renderers/hf.py:122] Traceback (most recent call last): [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:13:55 [renderers/hf.py:122] File "/opt/venv/lib/python3.12/site-packages/vllm/renderers/hf.py", line 120, in resolve_chat_template [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:13:55 [renderers/hf.py:122] return tokenizer.get_chat_template(chat_template, tools=tools) [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:13:55 [renderers/hf.py:122] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:13:55 [renderers/hf.py:122] File "/opt/venv/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 1825, in get_chat_template [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:13:55 [renderers/hf.py:122] raise ValueError( [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:13:55 [renderers/hf.py:122] ValueError: Cannot use chat template functions because tokenizer.chat_template is not set and no template argument was passed! For information about writing templates and setting the tokenizer.chat_template attribute, please see the documentation at https://huggingface.co/docs/transformers/main/en/chat_templating [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:13:55 [renderers/base.py:205] This model does not support chat template. [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 11:13:55 [entrypoints/openai/api_server.py:594] Starting vLLM server on https://0.0.0.0:8001 [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 11:13:55 [entrypoints/launcher.py:37] Available routes are: [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 11:13:55 [entrypoints/launcher.py:46] Route: /openapi.json, Methods: HEAD, GET [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 11:13:55 [entrypoints/launcher.py:46] Route: /docs, Methods: HEAD, GET [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 11:13:55 [entrypoints/launcher.py:46] Route: /docs/oauth2-redirect, Methods: HEAD, GET [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 11:13:55 [entrypoints/launcher.py:46] Route: /redoc, Methods: HEAD, GET [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 11:13:55 [entrypoints/launcher.py:46] Route: /tokenize, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 11:13:55 [entrypoints/launcher.py:46] Route: /detokenize, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 11:13:55 [entrypoints/launcher.py:46] Route: /load, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 11:13:55 [entrypoints/launcher.py:46] Route: /version, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 11:13:55 [entrypoints/launcher.py:46] Route: /health, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 11:13:55 [entrypoints/launcher.py:46] Route: /metrics, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 11:13:55 [entrypoints/launcher.py:46] Route: /v1/models, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 11:13:55 [entrypoints/launcher.py:46] Route: /ping, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 11:13:55 [entrypoints/launcher.py:46] Route: /ping, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 11:13:55 [entrypoints/launcher.py:46] Route: /invocations, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 11:13:55 [entrypoints/launcher.py:46] Route: /v1/chat/completions, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 11:13:55 [entrypoints/launcher.py:46] Route: /v1/chat/completions/batch, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 11:13:55 [entrypoints/launcher.py:46] Route: /v1/responses, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 11:13:55 [entrypoints/launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 11:13:55 [entrypoints/launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 11:13:55 [entrypoints/launcher.py:46] Route: /v1/completions, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 11:13:55 [entrypoints/launcher.py:46] Route: /v1/messages, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 11:13:55 [entrypoints/launcher.py:46] Route: /v1/messages/count_tokens, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 11:13:55 [entrypoints/launcher.py:46] Route: /inference/v1/generate, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 11:13:55 [entrypoints/launcher.py:46] Route: /scale_elastic_ep, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 11:13:55 [entrypoints/launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 11:13:55 [entrypoints/launcher.py:46] Route: /v1/chat/completions/render, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 11:13:55 [entrypoints/launcher.py:46] Route: /v1/completions/render, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO: Started server process [1] [e2e-llm-inference-service] (APIServer pid=1) INFO: Waiting for application startup. [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 11:13:55 [entrypoints/ssl.py:60] SSLCertRefresher monitors files: ['/var/run/kserve/tls/tls.key', '/var/run/kserve/tls/tls.crt'] [e2e-llm-inference-service] (APIServer pid=1) INFO: Application startup complete. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:13:57 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 11:13:57 [entrypoints/ssl.py:64] File change detected: modified - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 11:13:57 [entrypoints/ssl.py:34] Reloading SSL certificate chain [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 11:13:57 [entrypoints/ssl.py:64] File change detected: deleted - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 11:13:57 [entrypoints/ssl.py:34] Reloading SSL certificate chain [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 11:13:57 [entrypoints/ssl.py:64] File change detected: deleted - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 11:13:57 [entrypoints/ssl.py:34] Reloading SSL certificate chain [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 11:13:57 [entrypoints/ssl.py:64] File change detected: modified - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 11:13:57 [entrypoints/ssl.py:34] Reloading SSL certificate chain [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:14:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:14:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:14:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:14:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:14:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:14:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:15:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:15:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:15:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:15:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:15:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:15:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:16:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:16:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:16:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:16:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:16:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:16:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:17:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:17:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:17:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:17:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:17:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:17:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:18:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:18:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:18:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:18:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:18:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:18:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:19:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:19:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:19:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:19:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:19:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:19:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:20:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:20:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:20:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:20:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:20:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:20:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:21:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:21:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:21:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:21:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:21:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:21:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:22:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:22:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:22:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:22:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:22:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:22:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:23:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:23:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:23:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:23:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:23:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:23:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:24:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:24:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:24:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:24:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:24:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:24:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:25:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:25:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:25:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:25:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:25:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:25:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:26:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:26:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:26:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:26:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:26:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 11:26:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:172 ### Pod llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk (phase=Pending) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:212 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:228 # -- logs (current): unavailable ((400) [e2e-llm-inference-service] Reason: Bad Request [e2e-llm-inference-service] HTTP response headers: HTTPHeaderDict({'Audit-Id': '56713930-df99-4d9e-97b0-305d1a89e200', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'Date': 'Wed, 29 Jul 2026 11:27:02 GMT', 'Content-Length': '247'}) [e2e-llm-inference-service] HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"container \"main\" in pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk\" is waiting to start: ContainerCreating","reason":"BadRequest","code":400} [e2e-llm-inference-service] [e2e-llm-inference-service] ) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:172 ### Pod llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schexzxnd (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:212 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:225 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:226 {"level":"info","ts":1785323524.62708,"logger":"setup","caller":"runner/runner.go:196","msg":"GIE build","commit-sha":"unknown","build-ref":""} [e2e-llm-inference-service] {"level":"info","ts":1785323524.6272142,"logger":"setup","caller":"runner/runner.go:217","msg":"Flags processed","flags":{"cert-path":"/var/run/kserve/tls","config-file":"","config-text":"apiVersion: llm-d.ai/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n- type: disagg-headers-handler\n- type: prefill-filter\n- type: decode-filter\n- type: queue-scorer\n- type: kv-cache-utilization-scorer\n- type: active-request-scorer\n- type: prefix-cache-scorer\n- type: max-score-picker\n- type: always-disagg-pd-decider\n- parameters:\n deciders:\n prefill: always-disagg-pd-decider\n type: disagg-profile-handler\n- parameters:\n scheme: https\n type: metrics-data-source\nschedulingProfiles:\n- name: prefill\n plugins:\n - pluginRef: prefill-filter\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: queue-scorer\n weight: 2\n - pluginRef: kv-cache-utilization-scorer\n weight: 2\n - pluginRef: max-score-picker\n- name: decode\n plugins:\n - pluginRef: decode-filter\n - pluginRef: active-request-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n","disable-endpoint-subset-filter":false,"enable-cert-reload":true,"enable-grpc-stream-metrics":false,"enable-pprof":true,"endpoint-selector":"","endpoint-target-ports":{},"grpc-health-port":9003,"grpc-max-recv-msg-size":"","grpc-max-send-msg-size":"","grpc-port":9002,"ha-enable-leader-election":false,"health-checking":false,"metrics-endpoint-auth":true,"metrics-port":9090,"metrics-staleness-threshold":2000000000,"model-server-metrics-https-insecure-skip-verify":true,"model-server-metrics-path":"/metrics","model-server-metrics-port":0,"model-server-metrics-scheme":"http","pool-group":"inference.networking.k8s.io","pool-name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool","pool-namespace":"e2e-test-llm-inference-service-48639af5","refresh-metrics-interval":50000000,"refresh-prometheus-metrics-interval":5000000000,"secure-serving":true,"tracing":true,"v":2,"zap-devel":{},"zap-encoder":{},"zap-log-level":{},"zap-stacktrace-level":{},"zap-time-encoding":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785323524.6273978,"logger":"setup.trace","caller":"tracing/telemetry.go:123","msg":"init OTel trace exporter","type":"console"} [e2e-llm-inference-service] {"level":"info","ts":1785323524.628132,"caller":"loader/configloader.go:121","msg":"Loaded raw configuration","config":"{Plugins: [{Type: disagg-headers-handler} {Type: prefill-filter} {Type: decode-filter} {Type: queue-scorer} {Type: kv-cache-utilization-scorer} {Type: active-request-scorer} {Type: prefix-cache-scorer} {Type: max-score-picker} {Type: always-disagg-pd-decider} {Type: disagg-profile-handler, Parameters: {\"deciders\":{\"prefill\":\"always-disagg-pd-decider\"}}} {Type: metrics-data-source, Parameters: {\"scheme\":\"https\"}}], SchedulingProfiles: [{Name: prefill, Plugins: [{PluginRef: prefill-filter} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: queue-scorer, Weight: 2.00} {PluginRef: kv-cache-utilization-scorer, Weight: 2.00} {PluginRef: max-score-picker}]} {Name: decode, Plugins: [{PluginRef: decode-filter} {PluginRef: active-request-scorer, Weight: 2.00} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: max-score-picker}]}]}"} [e2e-llm-inference-service] {"level":"info","ts":1785323524.6281607,"logger":"setup","caller":"runner/runner.go:622","msg":"Data layer: ENABLED"} [e2e-llm-inference-service] {"level":"info","ts":1785323524.6284835,"logger":"setup","caller":"runner/runner.go:281","msg":"Raw config after phase one","config":{"apiVersion":"llm-d.ai/v1alpha1","dataLayer":null,"kind":"EndpointPickerConfig","plugins":[{"name":"disagg-headers-handler","parameters":null,"type":"disagg-headers-handler"},{"name":"prefill-filter","parameters":null,"type":"prefill-filter"},{"name":"decode-filter","parameters":null,"type":"decode-filter"},{"name":"queue-scorer","parameters":null,"type":"queue-scorer"},{"name":"kv-cache-utilization-scorer","parameters":null,"type":"kv-cache-utilization-scorer"},{"name":"active-request-scorer","parameters":null,"type":"active-request-scorer"},{"name":"prefix-cache-scorer","parameters":null,"type":"prefix-cache-scorer"},{"name":"max-score-picker","parameters":null,"type":"max-score-picker"},{"name":"always-disagg-pd-decider","parameters":null,"type":"always-disagg-pd-decider"},{"name":"disagg-profile-handler","parameters":{"deciders":{"prefill":"always-disagg-pd-decider"}},"type":"disagg-profile-handler"},{"name":"metrics-data-source","parameters":{"scheme":"https"},"type":"metrics-data-source"}],"schedulingProfiles":[{"name":"prefill","plugins":[{"pluginRef":"prefill-filter","weight":null},{"pluginRef":"prefix-cache-scorer","weight":3},{"pluginRef":"queue-scorer","weight":2},{"pluginRef":"kv-cache-utilization-scorer","weight":2},{"pluginRef":"max-score-picker","weight":null}]},{"name":"decode","plugins":[{"pluginRef":"decode-filter","weight":null},{"pluginRef":"active-request-scorer","weight":2},{"pluginRef":"prefix-cache-scorer","weight":3},{"pluginRef":"max-score-picker","weight":null}]}]}} [e2e-llm-inference-service] {"level":"info","ts":1785323524.6481562,"caller":"disagg/disagg_profile_handler.go:186","msg":"No deciders.encode configured, E disaggregation disabled"} [e2e-llm-inference-service] {"level":"info","ts":1785323524.6482568,"logger":"utilization-detector/utilization-detector","caller":"utilization/detector.go:83","msg":"Creating new UtilizationDetector","queueDepthThreshold":5,"kvCacheUtilThreshold":0.8,"metricsStalenessThreshold":"200ms","headroom":0} [e2e-llm-inference-service] {"level":"info","ts":1785323524.6483195,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"vllm","mapping":"Mapping{all specs enabled}"} [e2e-llm-inference-service] {"level":"info","ts":1785323524.6483588,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"sglang","mapping":"Mapping{disabled: [lora]}"} [e2e-llm-inference-service] {"level":"info","ts":1785323524.648393,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"trtllm-serve","mapping":"Mapping{disabled: [lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1785323524.6484585,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"triton-tensorrt-llm","mapping":"Mapping{disabled: [lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1785323524.6484807,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"triton","mapping":"Mapping{disabled: [kv, lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1785323524.6485484,"caller":"loader/configloader.go:154","msg":"Instantiated all plugins and applied system defaults. Effective raw configuration","config":"{Plugins: [{Name: disagg-headers-handler, Type: disagg-headers-handler} {Name: prefill-filter, Type: prefill-filter} {Name: decode-filter, Type: decode-filter} {Name: queue-scorer, Type: queue-scorer} {Name: kv-cache-utilization-scorer, Type: kv-cache-utilization-scorer} {Name: active-request-scorer, Type: active-request-scorer} {Name: prefix-cache-scorer, Type: prefix-cache-scorer} {Name: max-score-picker, Type: max-score-picker} {Name: always-disagg-pd-decider, Type: always-disagg-pd-decider} {Name: disagg-profile-handler, Type: disagg-profile-handler, Parameters: {\"deciders\":{\"prefill\":\"always-disagg-pd-decider\"}}} {Name: metrics-data-source, Type: metrics-data-source, Parameters: {\"scheme\":\"https\"}} {Name: fcfs-ordering-policy, Type: fcfs-ordering-policy} {Name: global-strict-fairness-policy, Type: global-strict-fairness-policy} {Name: static-usage-limit-policy, Type: static-usage-limit-policy} {Name: openai-parser, Type: openai-parser} {Name: anthropic-parser, Type: anthropic-parser} {Name: vllmhttp-parser, Type: vllmhttp-parser} {Name: utilization-detector, Type: utilization-detector} {Name: core-metrics-extractor, Type: core-metrics-extractor}], SchedulingProfiles: [{Name: prefill, Plugins: [{PluginRef: prefill-filter} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: queue-scorer, Weight: 2.00} {PluginRef: kv-cache-utilization-scorer, Weight: 2.00} {PluginRef: max-score-picker}]} {Name: decode, Plugins: [{PluginRef: decode-filter} {PluginRef: active-request-scorer, Weight: 2.00} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: max-score-picker}]}], DataLayer: {Sources: [{PluginRef: metrics-data-source, Extractors: [{PluginRef: core-metrics-extractor}]}], Discovery: }, FlowControl: {MaxBytes: unlimited, MaxRequests: unlimited, SaturationDetector: {PluginRef: utilization-detector}}, RequestHandler: {Parsers: [{PluginRef: openai-parser}, {PluginRef: anthropic-parser}, {PluginRef: vllmhttp-parser}]}}"} [e2e-llm-inference-service] {"level":"info","ts":1785323524.6486437,"caller":"datalayer/data_graph.go:116","msg":"auto-created default producer","producer":"inflight-load-producer/inflight-load-producer","dataKey":"InFlightLoadDataKey/inflight-load-producer","consumer":"active-request-scorer"} [e2e-llm-inference-service] {"level":"info","ts":1785323524.6486626,"caller":"approximateprefix/plugin.go:88","msg":"Prefix DataProducer initialized","config":{"autoTune":true,"blockSizeTokens":16,"blockSize":0,"maxPrefixBlocksToMatch":2048,"maxPrefixTokensToMatch":131072,"lruCapacityPerServer":31250}} [e2e-llm-inference-service] {"level":"info","ts":1785323524.6487682,"caller":"approximateprefix/plugin.go:111","msg":"WARNING: configured blockSizeTokens is below the recommended minimum, overriding it.","blockSizeTokens":16,"minimum":64,"issue":"https://github.com/llm-d/llm-d-router/issues/1158"} [e2e-llm-inference-service] {"level":"info","ts":1785323524.6487966,"caller":"datalayer/data_graph.go:116","msg":"auto-created default producer","producer":"approx-prefix-cache-producer/approx-prefix-cache-producer","dataKey":"PrefixCacheMatchInfoDataKey/approx-prefix-cache-producer","consumer":"disagg-profile-handler"} [e2e-llm-inference-service] {"level":"info","ts":1785323524.648812,"caller":"datalayer/data_graph.go:116","msg":"auto-created default producer","producer":"token-producer/token-producer","dataKey":"TokenizedPrompt/token-producer","consumer":"disagg-profile-handler"} [e2e-llm-inference-service] {"level":"info","ts":1785323524.6489372,"caller":"runner/runner.go:685","msg":"loaded configuration from file/text successfully"} [e2e-llm-inference-service] {"level":"info","ts":1785323524.6489472,"logger":"setup","caller":"runner/runner.go:308","msg":"EPP config after phase two","config":"{SchedulerConfig:{ProfileHandler: disagg-profile-handler/disagg-profile-handler, Profiles: map[decode:{Filters: [decode-filter/by-label], Scorers: [active-request-scorer/active-request-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000], Picker: max-score-picker/max-score-picker} prefill:{Filters: [prefill-filter/by-label], Scorers: [prefix-cache-scorer/prefix-cache-scorer: 3.000000, queue-scorer/queue-scorer: 2.000000, kv-cache-utilization-scorer/kv-cache-utilization-scorer: 2.000000], Picker: max-score-picker/max-score-picker}]} SaturationDetector:0xc000732d40 DataConfig:{Sources:[{Plugin:0xc000527290 Extractors:[0xc000732f40]}]} FlowControlConfig: ParserRegistry:0xc0007338c0}"} [e2e-llm-inference-service] {"level":"info","ts":1785323524.6691682,"logger":"setup","caller":"runner/runner.go:352","msg":"Setting pprof handlers"} [e2e-llm-inference-service] {"level":"info","ts":1785323524.6692033,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/mutex"} [e2e-llm-inference-service] {"level":"info","ts":1785323524.669221,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/"} [e2e-llm-inference-service] {"level":"info","ts":1785323524.6692264,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/cmdline"} [e2e-llm-inference-service] {"level":"info","ts":1785323524.669231,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/symbol"} [e2e-llm-inference-service] {"level":"info","ts":1785323524.6692357,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/trace"} [e2e-llm-inference-service] {"level":"info","ts":1785323524.6692398,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/goroutine"} [e2e-llm-inference-service] {"level":"info","ts":1785323524.6692445,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/allocs"} [e2e-llm-inference-service] {"level":"info","ts":1785323524.6692483,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/block"} [e2e-llm-inference-service] {"level":"info","ts":1785323524.6692529,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/profile"} [e2e-llm-inference-service] {"level":"info","ts":1785323524.669257,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/heap"} [e2e-llm-inference-service] {"level":"info","ts":1785323524.669262,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/threadcreate"} [e2e-llm-inference-service] {"level":"info","ts":1785323524.6692688,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/plugins/state"} [e2e-llm-inference-service] {"level":"info","ts":1785323524.6692731,"logger":"setup","caller":"runner/runner.go:373","msg":"parsed config","scheduler-config":"{ProfileHandler: disagg-profile-handler/disagg-profile-handler, Profiles: map[decode:{Filters: [decode-filter/by-label], Scorers: [active-request-scorer/active-request-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000], Picker: max-score-picker/max-score-picker} prefill:{Filters: [prefill-filter/by-label], Scorers: [prefix-cache-scorer/prefix-cache-scorer: 3.000000, queue-scorer/queue-scorer: 2.000000, kv-cache-utilization-scorer/kv-cache-utilization-scorer: 2.000000], Picker: max-score-picker/max-score-picker}]}"} [e2e-llm-inference-service] {"level":"info","ts":1785323524.66931,"logger":"setup","caller":"datalayer/runtime.go:99","msg":"Configuring datalayer runtime","numSources":1} [e2e-llm-inference-service] {"level":"info","ts":1785323524.6693199,"logger":"setup","caller":"datalayer/runtime.go:118","msg":"Processing source","source":"metrics-data-source","numExtractors":1} [e2e-llm-inference-service] {"level":"info","ts":1785323524.6693344,"logger":"setup","caller":"datalayer/runtime.go:147","msg":"Source configured","source":"metrics-data-source","extractors":["core-metrics-extractor/core-metrics-extractor"]} [e2e-llm-inference-service] {"level":"info","ts":1785323524.6693559,"logger":"setup","caller":"datalayer/runtime.go:206","msg":"Datalayer runtime configured","pollers":1,"notifiers":0,"endpointSources":1} [e2e-llm-inference-service] {"level":"info","ts":1785323524.6693704,"logger":"setup","caller":"runner/runner.go:833","msg":"Experimental Flow Control layer is disabled, using legacy admission control"} [e2e-llm-inference-service] {"level":"info","ts":1785323524.6694605,"logger":"setup","caller":"runner/runner.go:721","msg":"ExtProc server runner added to manager."} [e2e-llm-inference-service] {"level":"info","ts":1785323524.6694741,"logger":"setup","caller":"runner/runner.go:260","msg":"Controller manager starting"} [e2e-llm-inference-service] {"level":"info","ts":1785323524.6695015,"logger":"controller-runtime.metrics","caller":"server/server.go:208","msg":"Starting metrics server"} [e2e-llm-inference-service] {"level":"info","ts":1785323524.6697187,"caller":"runnable/grpc.go:35","msg":"gRPC server starting","name":"health"} [e2e-llm-inference-service] {"level":"info","ts":1785323524.669992,"caller":"runnable/grpc.go:43","msg":"gRPC server listening","name":"health","port":9003} [e2e-llm-inference-service] {"level":"info","ts":1785323524.6701174,"logger":"controller-runtime.metrics","caller":"server/server.go:247","msg":"Serving metrics server","bindAddress":":9090","secure":false} [e2e-llm-inference-service] {"level":"info","ts":1785323524.670263,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","source":"kind source: *v1.InferencePool"} [e2e-llm-inference-service] {"level":"info","ts":1785323524.670275,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferenceobjective","controllerGroup":"llm-d.ai","controllerKind":"InferenceObjective","source":"kind source: *v1alpha2.InferenceObjective"} [e2e-llm-inference-service] {"level":"info","ts":1785323524.670313,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"pod","controllerGroup":"","controllerKind":"Pod","source":"kind source: *v1.Pod"} [e2e-llm-inference-service] {"level":"info","ts":1785323524.6703331,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferencemodelrewrite","controllerGroup":"llm-d.ai","controllerKind":"InferenceModelRewrite","source":"kind source: *v1alpha2.InferenceModelRewrite"} [e2e-llm-inference-service] {"level":"info","ts":1785323524.6709101,"caller":"runnable/grpc.go:35","msg":"gRPC server starting","name":"ext-proc"} [e2e-llm-inference-service] {"level":"info","ts":1785323524.6709845,"caller":"runnable/grpc.go:43","msg":"gRPC server listening","name":"ext-proc","port":9002} [e2e-llm-inference-service] {"level":"info","ts":1785323524.6745045,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.InferenceObjective","reflector":"go/pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1785323524.6747441,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.InferenceModelRewrite","reflector":"go/pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1785323524.67495,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.InferencePool","reflector":"go/pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1785323524.675312,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Pod","reflector":"go/pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1785323524.7715075,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferencemodelrewrite","controllerGroup":"llm-d.ai","controllerKind":"InferenceModelRewrite"} [e2e-llm-inference-service] {"level":"info","ts":1785323524.7715364,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferencemodelrewrite","controllerGroup":"llm-d.ai","controllerKind":"InferenceModelRewrite","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1785323524.7714992,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferenceobjective","controllerGroup":"llm-d.ai","controllerKind":"InferenceObjective"} [e2e-llm-inference-service] {"level":"info","ts":1785323524.7715628,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferenceobjective","controllerGroup":"llm-d.ai","controllerKind":"InferenceObjective","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1785323524.771502,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool"} [e2e-llm-inference-service] {"level":"info","ts":1785323524.7715733,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1785323524.7717223,"caller":"controller/inferencepool_reconciler.go:46","msg":"Reconciling InferencePool","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool","namespace":"e2e-test-llm-inference-service-48639af5"},"namespace":"e2e-test-llm-inference-service-48639af5","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool","reconcileID":"6e5ea429-5a4d-4400-b274-10223e96a7c1"} [e2e-llm-inference-service] {"level":"info","ts":1785323524.8722558,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"pod","controllerGroup":"","controllerKind":"Pod"} [e2e-llm-inference-service] {"level":"info","ts":1785323524.8722975,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"pod","controllerGroup":"","controllerKind":"Pod","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1785323554.3596478,"caller":"controller/inferencepool_reconciler.go:46","msg":"Reconciling InferencePool","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool","namespace":"e2e-test-llm-inference-service-48639af5"},"namespace":"e2e-test-llm-inference-service-48639af5","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool","reconcileID":"420c74eb-e84a-402f-a45e-e9da284ff897"} [e2e-llm-inference-service] {"level":"info","ts":1785323637.837265,"caller":"inflightload/producer.go:308","msg":"Injected dynamic attribute into endpoint","key":"InFlightLoadDataKey/inflight-load-producer","endpoint":"e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67-rank-0"} [e2e-llm-inference-service] {"level":"info","ts":1785323637.8373055,"caller":"controller/pod_reconciler.go:99","msg":"Pod already exists","controller":"pod","controllerGroup":"","controllerKind":"Pod","Pod":{"name":"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67","namespace":"e2e-test-llm-inference-service-48639af5"},"namespace":"e2e-test-llm-inference-service-48639af5","name":"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67","reconcileID":"993d5c0e-3750-44e0-ade2-82fa8bf1bc9c"} [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-service [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 13130958-9e83-49fb-b734-cfafb2b34c54 [e2e-llm-inference-service] resourceVersion: '56732' [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:12:04Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-29T11:12:35Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-29T11:12:35Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - ip: 10.134.0.45 [e2e-llm-inference-service] nodeName: ip-10-0-140-240.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schexzxnd [e2e-llm-inference-service] uid: 701ba11c-e9ed-4ead-9b4f-39ec15efcd62 [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 262c8bf1-c542-41f2-899f-e05088c43f94 [e2e-llm-inference-service] resourceVersion: '57597' [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:12:03Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-29T11:13:57Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-29T11:13:57Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - ip: 10.132.0.61 [e2e-llm-inference-service] nodeName: ip-10-0-133-86.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67 [e2e-llm-inference-service] uid: b3e34fe2-65ed-4035-b463-759754ab7e02 [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67 [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: b3e34fe2-65ed-4035-b463-759754ab7e02 [e2e-llm-inference-service] resourceVersion: '57596' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:12:03Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 6d9fffb56c [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.132.0.61/23"],"mac_address":"0a:58:0a:84:00:3d","gateway_ips":["10.132.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.132.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.132.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.132.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.132.0.1"}],"ip_address":"10.132.0.61/23","gateway_ip":"10.132.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.132.0.61\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:84:00:3d\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c [e2e-llm-inference-service] uid: 6ef062ca-e5a1-4657-903b-13f97d39a25b [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-133-86 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-29T11:12:03Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-29T11:12:03Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"6ef062ca-e5a1-4657-903b-13f97d39a25b"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8001,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"llm-d-routing-sidecar"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAMESPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-29T11:12:07Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-29T11:13:57Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:initContainerStatuses: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.132.0.61"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] - name: kube-api-access-lvp74 [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/pd-sidecar [e2e-llm-inference-service] - --port=8000 [e2e-llm-inference-service] - --vllm-port=8001 [e2e-llm-inference-service] - --kv-connector=nixlv2 [e2e-llm-inference-service] - --enable-ssrf-protection=true [e2e-llm-inference-service] - --pool-group=inference.networking.x-k8s.io [e2e-llm-inference-service] - --inference-pool=e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] - --secure-proxy=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - --enable-tls=decoder [e2e-llm-inference-service] - --enable-tls=prefiller [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: INFERENCE_POOL_NAMESPACE [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] - name: INFERENCE_POOL_NAME [e2e-llm-inference-service] value: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kube-api-access-lvp74 [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000970000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to infer\ [e2e-llm-inference-service] \ RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null\n\ [e2e-llm-inference-service] \ grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/* 2>/dev/null\n\ [e2e-llm-inference-service] \n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"$hca_dir\"\ [e2e-llm-inference-service] \ ]; then\n hca_name=$(basename \"$hca_dir\")\n port_state_file=\"\ [e2e-llm-inference-service] $hca_dir/ports/1/state\" # Assume port 1\n type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\ [e2e-llm-inference-service] \n\n echo \"[Infer RoCE] Check if the port state file ${port_state_file}\ [e2e-llm-inference-service] \ exists and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] &&\ [e2e-llm-inference-service] \ grep -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found active\ [e2e-llm-inference-service] \ HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n else\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Skipping inactive or down HCA: $hca_name\"\ [e2e-llm-inference-service] \n fi\n fi\n done\n\n # Check if we found any active HCAs\n\ [e2e-llm-inference-service] \ if [ ${#active_hcas[@]} -gt 0 ]; then\n # Join the array elements with\ [e2e-llm-inference-service] \ a comma\n hca_port_pairs=()\n for hca in \"${active_hcas[@]}\";\ [e2e-llm-inference-service] \ do\n hca_port_pairs+=(\"${hca}:1\")\n done\n\n active_hca_list=$(IFS=,;\ [e2e-llm-inference-service] \ echo \"${active_hcas[*]}\")\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\"\ [e2e-llm-inference-service] )\n echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n\ [e2e-llm-inference-service] \ export UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found. NCCL_IB_HCA\ [e2e-llm-inference-service] \ will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Finding GID_INDEX for each active HCA (SR-IOV compatible)...\"\ [e2e-llm-inference-service] \n\n # For SR-IOV environments, find the most common IPv4 RoCE v2 GID index\ [e2e-llm-inference-service] \ across all HCAs\n declare -A gid_index_count\n declare -A hca_gid_index\n\ [e2e-llm-inference-service] \n for hca_name in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Processing HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for\ [e2e-llm-inference-service] \ this HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"$tpath\"\ [e2e-llm-inference-service] \ 2>/dev/null; then\n idx=$(basename \"$tpath\")\n \ [e2e-llm-inference-service] \ gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n \ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo \"\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Found IPv4 RoCE v2 GID for ${hca_name}:\ [e2e-llm-inference-service] \ index=${idx}, gid=${gid_value}\"\n hca_gid_index[\"${hca_name}\"\ [e2e-llm-inference-service] ]=\"${idx}\"\n gid_index_count[\"${idx}\"]=$((${gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]} + 1))\n break # Use first found IPv4 GID per\ [e2e-llm-inference-service] \ HCA\n fi\n fi\n done\n done\n\n\ [e2e-llm-inference-service] \ # Find the most common GID index (most likely to be consistent across\ [e2e-llm-inference-service] \ nodes)\n best_gid_index=\"\"\n max_count=0\n for idx in \"\ [e2e-llm-inference-service] ${!gid_index_count[@]}\"; do\n count=${gid_index_count[\"${idx}\"]}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n \ [e2e-llm-inference-service] \ if [ $count -gt $max_count ]; then\n max_count=$count\n\ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n #\ [e2e-llm-inference-service] \ Use deterministic fallback if tied - prefer index 3 (SR-IOV standard)\n \ [e2e-llm-inference-service] \ if [ ${#gid_index_count[@]} -gt 1 ]; then\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Multiple GID indices found, selecting most common: ${best_gid_index}\"\n \ [e2e-llm-inference-service] \ # If there's a tie, prefer index 3 as it's most common in SR-IOV setups\n\ [e2e-llm-inference-service] \ if [ -n \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\"\ [e2e-llm-inference-service] \ -eq \"$max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for NCCL,\ [e2e-llm-inference-service] \ NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR: No valid\ [e2e-llm-inference-service] \ IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any HCA.\"\n \ [e2e-llm-inference-service] \ fi\n else\n echo \"[Infer RoCE] No active HCAs found, skipping GID_INDEX\ [e2e-llm-inference-service] \ inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints landed in vLLM\ [e2e-llm-inference-service] \ 0.16.0 (vllm-project/vllm#30011).\n# Older versions still need the blanket\ [e2e-llm-inference-service] \ --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+ ]] &&\ [e2e-llm-inference-service] \ [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort -V | head\ [e2e-llm-inference-service] \ -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout 40\"\ [e2e-llm-inference-service] \nfi\n\n# --kv-transfer-config with OffloadingConnector requires vLLM 0.22.0+\ [e2e-llm-inference-service] \ (vllm-project/vllm#40020).\nKV_TRANSFER_ARGS=\"\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.22.0\" ]; then\n if [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\" != *\"\ [e2e-llm-inference-service] --kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"* ]] &&\ [e2e-llm-inference-service] \ [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n KV_TRANSFER_ARGS=\"\"\ [e2e-llm-inference-service] \n fi\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name \"\ [e2e-llm-inference-service] facebook/opt-125m\" \"publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8001 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n ${KV_TRANSFER_ARGS} \\\n \\\n --enable-ssl-refresh \\\n --ssl-certfile\ [e2e-llm-inference-service] \ /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8001 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-lvp74 [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000970000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] nodeName: ip-10-0-133-86.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c31,c20 [e2e-llm-inference-service] fsGroup: 1000970000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-mcnx5 [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-b9nll [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-29T11:12:08Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-29T11:12:08Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-29T11:13:57Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-29T11:13:57Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-29T11:12:03Z' [e2e-llm-inference-service] hostIP: 10.0.133.86 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.133.86 [e2e-llm-inference-service] podIP: 10.132.0.61 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.132.0.61 [e2e-llm-inference-service] startTime: '2026-07-29T11:12:03Z' [e2e-llm-inference-service] initContainerStatuses: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-07-29T11:12:07Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.9.0 [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-llm-d-router-disagg-sidecar@sha256:4d57f8fe65a63b44f345487af2b831e64ddf7fb01126d4f51cc8cf49b7412d00 [e2e-llm-inference-service] containerID: cri-o://35e8dddc73f5968744b03391a0f6b31ce1f5b556abb30bd1aecb24b97eb8ded9 [e2e-llm-inference-service] started: true [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-lvp74 [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000970000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000970000 [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-07-29T11:12:08Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: docker.io/vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] imageID: docker.io/vllm/vllm-openai-cpu@sha256:a8257c201fd2f696c146615987be61579aa2052367b73c27bd3fb1ab8e90bd4b [e2e-llm-inference-service] containerID: cri-o://281b111b443a2afe9c4a30e2a385a18b609a255b96b0f8155aa8dd8883616784 [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-lvp74 [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000970000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000970000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b89cd7957- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: db0a398f-6814-400e-bb81-fcae5bc445af [e2e-llm-inference-service] resourceVersion: '56195' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:12:03Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 7b89cd7957 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.133.0.58/23"],"mac_address":"0a:58:0a:85:00:3a","gateway_ips":["10.133.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.133.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.133.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.133.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.133.0.1"}],"ip_address":"10.133.0.58/23","gateway_ip":"10.133.0.1","role":"primary"}}' [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b89cd7957 [e2e-llm-inference-service] uid: 5fc57ba0-f26c-4cb6-a129-4a9835e30e32 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-129-80 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-29T11:12:03Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-29T11:12:03Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"5fc57ba0-f26c-4cb6-a129-4a9835e30e32"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-29T11:12:03Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] - name: kube-api-access-ttlhz [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to infer\ [e2e-llm-inference-service] \ RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null\n\ [e2e-llm-inference-service] \ grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/* 2>/dev/null\n\ [e2e-llm-inference-service] \n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"$hca_dir\"\ [e2e-llm-inference-service] \ ]; then\n hca_name=$(basename \"$hca_dir\")\n port_state_file=\"\ [e2e-llm-inference-service] $hca_dir/ports/1/state\" # Assume port 1\n type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\ [e2e-llm-inference-service] \n\n echo \"[Infer RoCE] Check if the port state file ${port_state_file}\ [e2e-llm-inference-service] \ exists and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] &&\ [e2e-llm-inference-service] \ grep -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found active\ [e2e-llm-inference-service] \ HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n else\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Skipping inactive or down HCA: $hca_name\"\ [e2e-llm-inference-service] \n fi\n fi\n done\n\n # Check if we found any active HCAs\n\ [e2e-llm-inference-service] \ if [ ${#active_hcas[@]} -gt 0 ]; then\n # Join the array elements with\ [e2e-llm-inference-service] \ a comma\n hca_port_pairs=()\n for hca in \"${active_hcas[@]}\";\ [e2e-llm-inference-service] \ do\n hca_port_pairs+=(\"${hca}:1\")\n done\n\n active_hca_list=$(IFS=,;\ [e2e-llm-inference-service] \ echo \"${active_hcas[*]}\")\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\"\ [e2e-llm-inference-service] )\n echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n\ [e2e-llm-inference-service] \ export UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found. NCCL_IB_HCA\ [e2e-llm-inference-service] \ will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Finding GID_INDEX for each active HCA (SR-IOV compatible)...\"\ [e2e-llm-inference-service] \n\n # For SR-IOV environments, find the most common IPv4 RoCE v2 GID index\ [e2e-llm-inference-service] \ across all HCAs\n declare -A gid_index_count\n declare -A hca_gid_index\n\ [e2e-llm-inference-service] \n for hca_name in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Processing HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for\ [e2e-llm-inference-service] \ this HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"$tpath\"\ [e2e-llm-inference-service] \ 2>/dev/null; then\n idx=$(basename \"$tpath\")\n \ [e2e-llm-inference-service] \ gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n \ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo \"\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Found IPv4 RoCE v2 GID for ${hca_name}:\ [e2e-llm-inference-service] \ index=${idx}, gid=${gid_value}\"\n hca_gid_index[\"${hca_name}\"\ [e2e-llm-inference-service] ]=\"${idx}\"\n gid_index_count[\"${idx}\"]=$((${gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]} + 1))\n break # Use first found IPv4 GID per\ [e2e-llm-inference-service] \ HCA\n fi\n fi\n done\n done\n\n\ [e2e-llm-inference-service] \ # Find the most common GID index (most likely to be consistent across\ [e2e-llm-inference-service] \ nodes)\n best_gid_index=\"\"\n max_count=0\n for idx in \"\ [e2e-llm-inference-service] ${!gid_index_count[@]}\"; do\n count=${gid_index_count[\"${idx}\"]}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n \ [e2e-llm-inference-service] \ if [ $count -gt $max_count ]; then\n max_count=$count\n\ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n #\ [e2e-llm-inference-service] \ Use deterministic fallback if tied - prefer index 3 (SR-IOV standard)\n \ [e2e-llm-inference-service] \ if [ ${#gid_index_count[@]} -gt 1 ]; then\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Multiple GID indices found, selecting most common: ${best_gid_index}\"\n \ [e2e-llm-inference-service] \ # If there's a tie, prefer index 3 as it's most common in SR-IOV setups\n\ [e2e-llm-inference-service] \ if [ -n \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\"\ [e2e-llm-inference-service] \ -eq \"$max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for NCCL,\ [e2e-llm-inference-service] \ NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR: No valid\ [e2e-llm-inference-service] \ IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any HCA.\"\n \ [e2e-llm-inference-service] \ fi\n else\n echo \"[Infer RoCE] No active HCAs found, skipping GID_INDEX\ [e2e-llm-inference-service] \ inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints landed in vLLM\ [e2e-llm-inference-service] \ 0.16.0 (vllm-project/vllm#30011).\n# Older versions still need the blanket\ [e2e-llm-inference-service] \ --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+ ]] &&\ [e2e-llm-inference-service] \ [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort -V | head\ [e2e-llm-inference-service] \ -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout 40\"\ [e2e-llm-inference-service] \nfi\n\n# --kv-transfer-config with OffloadingConnector requires vLLM 0.22.0+\ [e2e-llm-inference-service] \ (vllm-project/vllm#40020).\nKV_TRANSFER_ARGS=\"\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.22.0\" ]; then\n if [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\" != *\"\ [e2e-llm-inference-service] --kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"* ]] &&\ [e2e-llm-inference-service] \ [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n KV_TRANSFER_ARGS=\"\"\ [e2e-llm-inference-service] \n fi\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name \"\ [e2e-llm-inference-service] facebook/opt-125m\" \\\n --port 8000 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n ${KV_TRANSFER_ARGS} \\\n \\\n --enable-ssl-refresh \\\n --ssl-certfile\ [e2e-llm-inference-service] \ /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-ttlhz [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: File [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000970000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: default [e2e-llm-inference-service] serviceAccount: default [e2e-llm-inference-service] nodeName: ip-10-0-129-80.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c31,c20 [e2e-llm-inference-service] fsGroup: 1000970000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-mcnx5 [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Pending [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-29T11:12:03Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-29T11:12:03Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-29T11:12:03Z' [e2e-llm-inference-service] reason: ContainersNotReady [e2e-llm-inference-service] message: 'containers with unready status: [main]' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-29T11:12:03Z' [e2e-llm-inference-service] reason: ContainersNotReady [e2e-llm-inference-service] message: 'containers with unready status: [main]' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-29T11:12:03Z' [e2e-llm-inference-service] hostIP: 10.0.129.80 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.129.80 [e2e-llm-inference-service] startTime: '2026-07-29T11:12:03Z' [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] waiting: [e2e-llm-inference-service] reason: ContainerCreating [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: false [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] imageID: '' [e2e-llm-inference-service] started: false [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-ttlhz [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schexzxnd [e2e-llm-inference-service] generateName: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-7fc69df7b8- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 701ba11c-e9ed-4ead-9b4f-39ec15efcd62 [e2e-llm-inference-service] resourceVersion: '56731' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:12:04Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 7fc69df7b8 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.10.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.134.0.45/23"],"mac_address":"0a:58:0a:86:00:2d","gateway_ips":["10.134.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.134.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.134.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.134.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.134.0.1"}],"ip_address":"10.134.0.45/23","gateway_ip":"10.134.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.134.0.45\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:86:00:2d\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-7fc69df7b8 [e2e-llm-inference-service] uid: 7ff489ab-d9d8-480c-8dd5-fb288b8b1922 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-140-240 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-29T11:12:04Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-29T11:12:04Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7ff489ab-d9d8-480c-8dd5-fb288b8b1922"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-29T11:12:04Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-29T11:12:35Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.134.0.45"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kube-api-access-snncn [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: llm-d.ai/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n- type:\ [e2e-llm-inference-service] \ disagg-headers-handler\n- type: prefill-filter\n- type: decode-filter\n- type:\ [e2e-llm-inference-service] \ queue-scorer\n- type: kv-cache-utilization-scorer\n- type: active-request-scorer\n\ [e2e-llm-inference-service] - type: prefix-cache-scorer\n- type: max-score-picker\n- type: always-disagg-pd-decider\n\ [e2e-llm-inference-service] - parameters:\n deciders:\n prefill: always-disagg-pd-decider\n type:\ [e2e-llm-inference-service] \ disagg-profile-handler\n- parameters:\n scheme: https\n type: metrics-data-source\n\ [e2e-llm-inference-service] schedulingProfiles:\n- name: prefill\n plugins:\n - pluginRef: prefill-filter\n\ [e2e-llm-inference-service] \ - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: queue-scorer\n\ [e2e-llm-inference-service] \ weight: 2\n - pluginRef: kv-cache-utilization-scorer\n weight: 2\n\ [e2e-llm-inference-service] \ - pluginRef: max-score-picker\n- name: decode\n plugins:\n - pluginRef:\ [e2e-llm-inference-service] \ decode-filter\n - pluginRef: active-request-scorer\n weight: 2\n - pluginRef:\ [e2e-llm-inference-service] \ prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kube-api-access-snncn [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000970000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] nodeName: ip-10-0-140-240.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c31,c20 [e2e-llm-inference-service] fsGroup: 1000970000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-mcnx5 [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-p7mgs [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-29T11:12:04Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-29T11:12:04Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-29T11:12:35Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-29T11:12:35Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-29T11:12:04Z' [e2e-llm-inference-service] hostIP: 10.0.140.240 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.140.240 [e2e-llm-inference-service] podIP: 10.134.0.45 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.134.0.45 [e2e-llm-inference-service] startTime: '2026-07-29T11:12:04Z' [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-07-29T11:12:04Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-llm-d-router-endpoint-picker@sha256:7fd1d61cb5505e026eebab0b9653758779c3dee241e37bea92a0bab961a48c50 [e2e-llm-inference-service] containerID: cri-o://293e7d4ea2b49e05abc2d09d4b79ab8ce550056d0a684ab3c79924c3ffabee8c [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-snncn [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000970000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000970000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: d71b7906-0bba-43e7-8726-7c49f4f00135 [e2e-llm-inference-service] resourceVersion: '56187' [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:12:03Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] openshift.io/internal-registry-pull-secret-ref: llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-p7mgs [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 7031a75f-c7bb-4cfb-921d-6fc95e619eb3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: openshift.io/image-registry-pull-secrets_service-account-controller [e2e-llm-inference-service] operation: Apply [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-29T11:12:03Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:imagePullSecrets: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:openshift.io/internal-registry-pull-secret-ref: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] k:{"name":"llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-p7mgs"}: {} [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-29T11:12:03Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7031a75f-c7bb-4cfb-921d-6fc95e619eb3"}: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"default-dockercfg-mcnx5"}: {} [e2e-llm-inference-service] k:{"name":"seaweedfs-s3-creds"}: {} [e2e-llm-inference-service] secrets: [e2e-llm-inference-service] - name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: default-dockercfg-mcnx5 [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-p7mgs [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-mcnx5 [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-p7mgs [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: ServiceAccount [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: d5f6adcd-478c-4109-9a4e-fc66215c6c75 [e2e-llm-inference-service] resourceVersion: '56144' [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:12:03Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] openshift.io/internal-registry-pull-secret-ref: llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-b9nll [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 7031a75f-c7bb-4cfb-921d-6fc95e619eb3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: openshift.io/image-registry-pull-secrets_service-account-controller [e2e-llm-inference-service] operation: Apply [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-29T11:12:03Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:imagePullSecrets: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:openshift.io/internal-registry-pull-secret-ref: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] k:{"name":"llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-b9nll"}: {} [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-29T11:12:03Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7031a75f-c7bb-4cfb-921d-6fc95e619eb3"}: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"default-dockercfg-mcnx5"}: {} [e2e-llm-inference-service] k:{"name":"seaweedfs-s3-creds"}: {} [e2e-llm-inference-service] secrets: [e2e-llm-inference-service] - name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: default-dockercfg-mcnx5 [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-b9nll [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-mcnx5 [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-b9nll [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: ServiceAccount [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-service [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 37df8ce9-2999-41e5-bfb3-5330414daa39 [e2e-llm-inference-service] resourceVersion: '56213' [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:12:04Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 7031a75f-c7bb-4cfb-921d-6fc95e619eb3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-29T11:12:04Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7031a75f-c7bb-4cfb-921d-6fc95e619eb3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] targetPort: grpc [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] targetPort: grpc-health [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] targetPort: metrics [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] targetPort: zmq [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] clusterIP: 172.31.233.132 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.233.132 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: a097081c-e89d-4216-8f0d-905fea115c6c [e2e-llm-inference-service] resourceVersion: '56173' [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:12:03Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 7031a75f-c7bb-4cfb-921d-6fc95e619eb3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-29T11:12:03Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7031a75f-c7bb-4cfb-921d-6fc95e619eb3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] targetPort: 8000 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] clusterIP: 172.31.1.230 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.1.230 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: a6fd7e31-597f-4477-9b06-4ec4c6fc3d20 [e2e-llm-inference-service] resourceVersion: '57602' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:12:03Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 7031a75f-c7bb-4cfb-921d-6fc95e619eb3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-29T11:12:03Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7031a75f-c7bb-4cfb-921d-6fc95e619eb3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:rollingUpdate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:maxSurge: {} [e2e-llm-inference-service] f:maxUnavailable: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8001,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"llm-d-routing-sidecar"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAMESPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-29T11:13:57Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/pd-sidecar [e2e-llm-inference-service] - --port=8000 [e2e-llm-inference-service] - --vllm-port=8001 [e2e-llm-inference-service] - --kv-connector=nixlv2 [e2e-llm-inference-service] - --enable-ssrf-protection=true [e2e-llm-inference-service] - --pool-group=inference.networking.x-k8s.io [e2e-llm-inference-service] - --inference-pool=e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] - --secure-proxy=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - --enable-tls=decoder [e2e-llm-inference-service] - --enable-tls=prefiller [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: INFERENCE_POOL_NAMESPACE [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] - name: INFERENCE_POOL_NAME [e2e-llm-inference-service] value: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n # Check if\ [e2e-llm-inference-service] \ we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ # Join the array elements with a comma\n hca_port_pairs=()\n \ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n hca_port_pairs+=(\"\ [e2e-llm-inference-service] ${hca}:1\")\n done\n\n active_hca_list=$(IFS=,; echo \"${active_hcas[*]}\"\ [e2e-llm-inference-service] )\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\n# --kv-transfer-config with OffloadingConnector requires vLLM\ [e2e-llm-inference-service] \ 0.22.0+ (vllm-project/vllm#40020).\nKV_TRANSFER_ARGS=\"\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.22.0\" ]; then\n if [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\" !=\ [e2e-llm-inference-service] \ *\"--kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"\ [e2e-llm-inference-service] * ]] && [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n KV_TRANSFER_ARGS=\"\ [e2e-llm-inference-service] \"\n fi\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \"publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8001 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n ${KV_TRANSFER_ARGS} \\\n \\\n --enable-ssl-refresh \\\n --ssl-certfile\ [e2e-llm-inference-service] \ /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8001 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: RollingUpdate [e2e-llm-inference-service] rollingUpdate: [e2e-llm-inference-service] maxUnavailable: 25% [e2e-llm-inference-service] maxSurge: 25% [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-29T11:13:57Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-29T11:13:57Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-29T11:13:57Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-29T11:12:03Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c" [e2e-llm-inference-service] has successfully progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 57ddeb3a-fb68-4fbf-83f4-ee04189254ee [e2e-llm-inference-service] resourceVersion: '62541' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:12:03Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 7031a75f-c7bb-4cfb-921d-6fc95e619eb3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-29T11:12:03Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7031a75f-c7bb-4cfb-921d-6fc95e619eb3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:rollingUpdate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:maxSurge: {} [e2e-llm-inference-service] f:maxUnavailable: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-29T11:22:04Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:unavailableReplicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n # Check if\ [e2e-llm-inference-service] \ we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ # Join the array elements with a comma\n hca_port_pairs=()\n \ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n hca_port_pairs+=(\"\ [e2e-llm-inference-service] ${hca}:1\")\n done\n\n active_hca_list=$(IFS=,; echo \"${active_hcas[*]}\"\ [e2e-llm-inference-service] )\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\n# --kv-transfer-config with OffloadingConnector requires vLLM\ [e2e-llm-inference-service] \ 0.22.0+ (vllm-project/vllm#40020).\nKV_TRANSFER_ARGS=\"\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.22.0\" ]; then\n if [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\" !=\ [e2e-llm-inference-service] \ *\"--kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"\ [e2e-llm-inference-service] * ]] && [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n KV_TRANSFER_ARGS=\"\ [e2e-llm-inference-service] \"\n fi\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \\\n --port 8000 \\\n ${ACCESS_LOG_ARGS} \\\n\ [e2e-llm-inference-service] \ ${SHUTDOWN_TIMEOUT_ARGS} \\\n ${KV_TRANSFER_ARGS} \\\n \\\n --enable-ssl-refresh\ [e2e-llm-inference-service] \ \\\n --ssl-certfile /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: File [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: RollingUpdate [e2e-llm-inference-service] rollingUpdate: [e2e-llm-inference-service] maxUnavailable: 25% [e2e-llm-inference-service] maxSurge: 25% [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] unavailableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastUpdateTime: '2026-07-29T11:12:03Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-29T11:12:03Z' [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastUpdateTime: '2026-07-29T11:22:04Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-29T11:22:04Z' [e2e-llm-inference-service] reason: ProgressDeadlineExceeded [e2e-llm-inference-service] message: ReplicaSet "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b89cd7957" [e2e-llm-inference-service] has timed out progressing. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 460f7a93-5545-4102-b670-8778a4c4e610 [e2e-llm-inference-service] resourceVersion: '56736' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:12:04Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 7031a75f-c7bb-4cfb-921d-6fc95e619eb3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-29T11:12:04Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7031a75f-c7bb-4cfb-921d-6fc95e619eb3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-29T11:12:35Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.10.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: llm-d.ai/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n-\ [e2e-llm-inference-service] \ type: disagg-headers-handler\n- type: prefill-filter\n- type: decode-filter\n\ [e2e-llm-inference-service] - type: queue-scorer\n- type: kv-cache-utilization-scorer\n- type: active-request-scorer\n\ [e2e-llm-inference-service] - type: prefix-cache-scorer\n- type: max-score-picker\n- type: always-disagg-pd-decider\n\ [e2e-llm-inference-service] - parameters:\n deciders:\n prefill: always-disagg-pd-decider\n\ [e2e-llm-inference-service] \ type: disagg-profile-handler\n- parameters:\n scheme: https\n type:\ [e2e-llm-inference-service] \ metrics-data-source\nschedulingProfiles:\n- name: prefill\n plugins:\n\ [e2e-llm-inference-service] \ - pluginRef: prefill-filter\n - pluginRef: prefix-cache-scorer\n \ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: queue-scorer\n weight: 2\n - pluginRef:\ [e2e-llm-inference-service] \ kv-cache-utilization-scorer\n weight: 2\n - pluginRef: max-score-picker\n\ [e2e-llm-inference-service] - name: decode\n plugins:\n - pluginRef: decode-filter\n - pluginRef:\ [e2e-llm-inference-service] \ active-request-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n\ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: Recreate [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-29T11:12:35Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-29T11:12:35Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-29T11:12:35Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-29T11:12:04Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-7fc69df7b8" [e2e-llm-inference-service] has successfully progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 6ef062ca-e5a1-4657-903b-13f97d39a25b [e2e-llm-inference-service] resourceVersion: '57601' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:12:03Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 6d9fffb56c [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '2' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] uid: a6fd7e31-597f-4477-9b06-4ec4c6fc3d20 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-29T11:12:03Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"a6fd7e31-597f-4477-9b06-4ec4c6fc3d20"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8001,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"llm-d-routing-sidecar"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAMESPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-29T11:13:57Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 6d9fffb56c [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 6d9fffb56c [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/pd-sidecar [e2e-llm-inference-service] - --port=8000 [e2e-llm-inference-service] - --vllm-port=8001 [e2e-llm-inference-service] - --kv-connector=nixlv2 [e2e-llm-inference-service] - --enable-ssrf-protection=true [e2e-llm-inference-service] - --pool-group=inference.networking.x-k8s.io [e2e-llm-inference-service] - --inference-pool=e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] - --secure-proxy=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - --enable-tls=decoder [e2e-llm-inference-service] - --enable-tls=prefiller [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: INFERENCE_POOL_NAMESPACE [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] - name: INFERENCE_POOL_NAME [e2e-llm-inference-service] value: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n # Check if\ [e2e-llm-inference-service] \ we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ # Join the array elements with a comma\n hca_port_pairs=()\n \ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n hca_port_pairs+=(\"\ [e2e-llm-inference-service] ${hca}:1\")\n done\n\n active_hca_list=$(IFS=,; echo \"${active_hcas[*]}\"\ [e2e-llm-inference-service] )\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\n# --kv-transfer-config with OffloadingConnector requires vLLM\ [e2e-llm-inference-service] \ 0.22.0+ (vllm-project/vllm#40020).\nKV_TRANSFER_ARGS=\"\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.22.0\" ]; then\n if [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\" !=\ [e2e-llm-inference-service] \ *\"--kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"\ [e2e-llm-inference-service] * ]] && [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n KV_TRANSFER_ARGS=\"\ [e2e-llm-inference-service] \"\n fi\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \"publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8001 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n ${KV_TRANSFER_ARGS} \\\n \\\n --enable-ssl-refresh \\\n --ssl-certfile\ [e2e-llm-inference-service] \ /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8001 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b89cd7957 [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 5fc57ba0-f26c-4cb6-a129-4a9835e30e32 [e2e-llm-inference-service] resourceVersion: '56188' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:12:03Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 7b89cd7957 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '2' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill [e2e-llm-inference-service] uid: 57ddeb3a-fb68-4fbf-83f4-ee04189254ee [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-29T11:12:03Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"57ddeb3a-fb68-4fbf-83f4-ee04189254ee"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-29T11:12:03Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 7b89cd7957 [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 7b89cd7957 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n # Check if\ [e2e-llm-inference-service] \ we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ # Join the array elements with a comma\n hca_port_pairs=()\n \ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n hca_port_pairs+=(\"\ [e2e-llm-inference-service] ${hca}:1\")\n done\n\n active_hca_list=$(IFS=,; echo \"${active_hcas[*]}\"\ [e2e-llm-inference-service] )\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\n# --kv-transfer-config with OffloadingConnector requires vLLM\ [e2e-llm-inference-service] \ 0.22.0+ (vllm-project/vllm#40020).\nKV_TRANSFER_ARGS=\"\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.22.0\" ]; then\n if [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\" !=\ [e2e-llm-inference-service] \ *\"--kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"\ [e2e-llm-inference-service] * ]] && [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n KV_TRANSFER_ARGS=\"\ [e2e-llm-inference-service] \"\n fi\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \\\n --port 8000 \\\n ${ACCESS_LOG_ARGS} \\\n\ [e2e-llm-inference-service] \ ${SHUTDOWN_TIMEOUT_ARGS} \\\n ${KV_TRANSFER_ARGS} \\\n \\\n --enable-ssl-refresh\ [e2e-llm-inference-service] \ \\\n --ssl-certfile /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: File [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-7fc69df7b8 [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 7ff489ab-d9d8-480c-8dd5-fb288b8b1922 [e2e-llm-inference-service] resourceVersion: '56735' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:12:04Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 7fc69df7b8 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler [e2e-llm-inference-service] uid: 460f7a93-5545-4102-b670-8778a4c4e610 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-29T11:12:04Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"460f7a93-5545-4102-b670-8778a4c4e610"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-29T11:12:35Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 7fc69df7b8 [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 7fc69df7b8 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.10.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: llm-d.ai/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n-\ [e2e-llm-inference-service] \ type: disagg-headers-handler\n- type: prefill-filter\n- type: decode-filter\n\ [e2e-llm-inference-service] - type: queue-scorer\n- type: kv-cache-utilization-scorer\n- type: active-request-scorer\n\ [e2e-llm-inference-service] - type: prefix-cache-scorer\n- type: max-score-picker\n- type: always-disagg-pd-decider\n\ [e2e-llm-inference-service] - parameters:\n deciders:\n prefill: always-disagg-pd-decider\n\ [e2e-llm-inference-service] \ type: disagg-profile-handler\n- parameters:\n scheme: https\n type:\ [e2e-llm-inference-service] \ metrics-data-source\nschedulingProfiles:\n- name: prefill\n plugins:\n\ [e2e-llm-inference-service] \ - pluginRef: prefill-filter\n - pluginRef: prefix-cache-scorer\n \ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: queue-scorer\n weight: 2\n - pluginRef:\ [e2e-llm-inference-service] \ kv-cache-utilization-scorer\n weight: 2\n - pluginRef: max-score-picker\n\ [e2e-llm-inference-service] - name: decode\n plugins:\n - pluginRef: decode-filter\n - pluginRef:\ [e2e-llm-inference-service] \ active-request-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n\ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-rb [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 4b044160-a2be-4444-98e0-c3b88e9ffb4f [e2e-llm-inference-service] resourceVersion: '56204' [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:12:04Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 7031a75f-c7bb-4cfb-921d-6fc95e619eb3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-29T11:12:03Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7031a75f-c7bb-4cfb-921d-6fc95e619eb3"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] apiGroup: rbac.authorization.k8s.io [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-role [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-rb [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 15a38e13-c8cb-4dae-83cc-4deaf0d483c2 [e2e-llm-inference-service] resourceVersion: '56154' [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:12:03Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 7031a75f-c7bb-4cfb-921d-6fc95e619eb3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-29T11:12:03Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7031a75f-c7bb-4cfb-921d-6fc95e619eb3"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] apiGroup: rbac.authorization.k8s.io [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-role [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-role [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: d5e3c342-0019-4844-ba6a-db60ad2f9a6d [e2e-llm-inference-service] resourceVersion: '56200' [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:12:03Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 7031a75f-c7bb-4cfb-921d-6fc95e619eb3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-29T11:12:03Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7031a75f-c7bb-4cfb-921d-6fc95e619eb3"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - llm-d.ai [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] - create [e2e-llm-inference-service] - update [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - delete [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-role [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 24faec35-de32-46dd-9f5e-ad06b2cc293a [e2e-llm-inference-service] resourceVersion: '56146' [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:12:03Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 7031a75f-c7bb-4cfb-921d-6fc95e619eb3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-29T11:12:03Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7031a75f-c7bb-4cfb-921d-6fc95e619eb3"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-service-4bb28 [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-e8706282-epp-service- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 4dbaefd0-b3cb-4d1d-bda8-e322ad9c5146 [e2e-llm-inference-service] resourceVersion: '56733' [e2e-llm-inference-service] generation: 3 [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:12:04Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: llmisvc-model-pvc-router-manage-e8706282-epp-service [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-29T11:12:35Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-service [e2e-llm-inference-service] uid: 37df8ce9-2999-41e5-bfb3-5330414daa39 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-29T11:12:35Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"37df8ce9-2999-41e5-bfb3-5330414daa39"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.134.0.45 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schexzxnd [e2e-llm-inference-service] uid: 701ba11c-e9ed-4ead-9b4f-39ec15efcd62 [e2e-llm-inference-service] nodeName: ip-10-0-140-240.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-ss6rvm [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 4a081b35-b6aa-4c53-9aad-68fd54a1ba0f [e2e-llm-inference-service] resourceVersion: '57600' [e2e-llm-inference-service] generation: 3 [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:12:03Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-29T11:13:57Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] uid: a097081c-e89d-4216-8f0d-905fea115c6c [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-29T11:13:57Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"a097081c-e89d-4216-8f0d-905fea115c6c"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.132.0.61 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67 [e2e-llm-inference-service] uid: b3e34fe2-65ed-4035-b463-759754ab7e02 [e2e-llm-inference-service] nodeName: ip-10-0-133-86.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-rb [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 4b044160-a2be-4444-98e0-c3b88e9ffb4f [e2e-llm-inference-service] resourceVersion: '56204' [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:12:04Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 7031a75f-c7bb-4cfb-921d-6fc95e619eb3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-29T11:12:03Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7031a75f-c7bb-4cfb-921d-6fc95e619eb3"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] userNames: [e2e-llm-inference-service] - system:serviceaccount:e2e-test-llm-inference-service-48639af5:llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] groupNames: null [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-role [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-rb [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 15a38e13-c8cb-4dae-83cc-4deaf0d483c2 [e2e-llm-inference-service] resourceVersion: '56154' [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:12:03Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 7031a75f-c7bb-4cfb-921d-6fc95e619eb3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-29T11:12:03Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7031a75f-c7bb-4cfb-921d-6fc95e619eb3"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] userNames: [e2e-llm-inference-service] - system:serviceaccount:e2e-test-llm-inference-service-48639af5:llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] groupNames: null [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-role [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-role [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: d5e3c342-0019-4844-ba6a-db60ad2f9a6d [e2e-llm-inference-service] resourceVersion: '56200' [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:12:03Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 7031a75f-c7bb-4cfb-921d-6fc95e619eb3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-29T11:12:03Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7031a75f-c7bb-4cfb-921d-6fc95e619eb3"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - llm-d.ai [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - create [e2e-llm-inference-service] - delete [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - update [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-role [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 24faec35-de32-46dd-9f5e-ad06b2cc293a [e2e-llm-inference-service] resourceVersion: '56146' [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:12:03Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 7031a75f-c7bb-4cfb-921d-6fc95e619eb3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-29T11:12:03Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7031a75f-c7bb-4cfb-921d-6fc95e619eb3"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:12:04Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7031a75f-c7bb-4cfb-921d-6fc95e619eb3"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-29T11:12:34Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-29T11:12:34Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-29T11:12:35Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-route [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 7031a75f-c7bb-4cfb-921d-6fc95e619eb3 [e2e-llm-inference-service] resourceVersion: '56722' [e2e-llm-inference-service] uid: af1463f3-0447-4367-935e-b6502d1afdae [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-29T11:12:04Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-29T11:12:34Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-29T11:12:04Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:12:04Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7031a75f-c7bb-4cfb-921d-6fc95e619eb3"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-29T11:12:34Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-29T11:12:34Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-29T11:12:35Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-route [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 7031a75f-c7bb-4cfb-921d-6fc95e619eb3 [e2e-llm-inference-service] resourceVersion: '56722' [e2e-llm-inference-service] uid: af1463f3-0447-4367-935e-b6502d1afdae [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-29T11:12:04Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-29T11:12:34Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-29T11:12:04Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:12:04Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7031a75f-c7bb-4cfb-921d-6fc95e619eb3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:endpointPickerRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:number: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:matchLabels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPorts: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-29T11:12:04Z' [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-29T11:12:34Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 7031a75f-c7bb-4cfb-921d-6fc95e619eb3 [e2e-llm-inference-service] resourceVersion: '56705' [e2e-llm-inference-service] uid: 1776dcd4-209c-48f2-b643-812d61e42746 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] appProtocol: http [e2e-llm-inference-service] endpointPickerRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-service [e2e-llm-inference-service] port: [e2e-llm-inference-service] number: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPorts: [e2e-llm-inference-service] - number: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-29T11:12:34Z' [e2e-llm-inference-service] message: Referenced by an HTTPRoute accepted by the parentRef Gateway [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-29T11:12:34Z' [e2e-llm-inference-service] message: Referenced ExtensionRef resolved successfully [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: networking.istio.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] kind: AuthPolicy [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:12:06Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-policies [e2e-llm-inference-service] app.kubernetes.io/managed-by: odh-model-controller [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7031a75f-c7bb-4cfb-921d-6fc95e619eb3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:rules: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:authentication: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:public: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:anonymous: {} [e2e-llm-inference-service] f:credentials: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:overrides: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fairness: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:response: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:success: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:headers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:x-gateway-inference-fairness-id: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:x-gateway-inference-objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:targetRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-29T11:12:06Z' [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Accepted"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Enforced"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-29T11:12:07Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-route-authn [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 7031a75f-c7bb-4cfb-921d-6fc95e619eb3 [e2e-llm-inference-service] resourceVersion: '56400' [e2e-llm-inference-service] uid: a57829c2-58aa-449e-827c-b1745288c7b4 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] rules: [e2e-llm-inference-service] authentication: [e2e-llm-inference-service] public: [e2e-llm-inference-service] anonymous: {} [e2e-llm-inference-service] credentials: {} [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] overrides: [e2e-llm-inference-service] fairness: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] objective: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] response: [e2e-llm-inference-service] success: [e2e-llm-inference-service] headers: [e2e-llm-inference-service] x-gateway-inference-fairness-id: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.fairness [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] x-gateway-inference-objective: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.objective [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-route [e2e-llm-inference-service] status: [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-29T11:12:07Z' [e2e-llm-inference-service] message: AuthPolicy has been accepted [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-29T11:12:07Z' [e2e-llm-inference-service] message: AuthPolicy has been successfully enforced [e2e-llm-inference-service] reason: Enforced [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Enforced [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:12:04Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7031a75f-c7bb-4cfb-921d-6fc95e619eb3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-29T11:12:04Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-scheduler [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 7031a75f-c7bb-4cfb-921d-6fc95e619eb3 [e2e-llm-inference-service] resourceVersion: '56283' [e2e-llm-inference-service] uid: 57f7e0af-7498-4748-bf2d-44962a2f5156 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-epp-service.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-epp-service.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:12:34Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7031a75f-c7bb-4cfb-921d-6fc95e619eb3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-29T11:12:34Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-shadow-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 7031a75f-c7bb-4cfb-921d-6fc95e619eb3 [e2e-llm-inference-service] resourceVersion: '56715' [e2e-llm-inference-service] uid: e4bd422b-1780-4895-95f7-efa35b6c399c [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-inference--ip-8bbb5aea.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:12:05Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7031a75f-c7bb-4cfb-921d-6fc95e619eb3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-29T11:12:05Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 7031a75f-c7bb-4cfb-921d-6fc95e619eb3 [e2e-llm-inference-service] resourceVersion: '56303' [e2e-llm-inference-service] uid: d55aa8a6-233d-4ed3-b2b3-345c39e49fb0 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:12:04Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7031a75f-c7bb-4cfb-921d-6fc95e619eb3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-29T11:12:04Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-scheduler [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 7031a75f-c7bb-4cfb-921d-6fc95e619eb3 [e2e-llm-inference-service] resourceVersion: '56283' [e2e-llm-inference-service] uid: 57f7e0af-7498-4748-bf2d-44962a2f5156 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-epp-service.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-epp-service.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:12:34Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7031a75f-c7bb-4cfb-921d-6fc95e619eb3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-29T11:12:34Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-shadow-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 7031a75f-c7bb-4cfb-921d-6fc95e619eb3 [e2e-llm-inference-service] resourceVersion: '56715' [e2e-llm-inference-service] uid: e4bd422b-1780-4895-95f7-efa35b6c399c [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-inference--ip-8bbb5aea.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:12:05Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7031a75f-c7bb-4cfb-921d-6fc95e619eb3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-29T11:12:05Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 7031a75f-c7bb-4cfb-921d-6fc95e619eb3 [e2e-llm-inference-service] resourceVersion: '56303' [e2e-llm-inference-service] uid: d55aa8a6-233d-4ed3-b2b3-345c39e49fb0 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:12:04Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7031a75f-c7bb-4cfb-921d-6fc95e619eb3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-29T11:12:04Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-scheduler [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 7031a75f-c7bb-4cfb-921d-6fc95e619eb3 [e2e-llm-inference-service] resourceVersion: '56283' [e2e-llm-inference-service] uid: 57f7e0af-7498-4748-bf2d-44962a2f5156 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-epp-service.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-epp-service.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:12:34Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7031a75f-c7bb-4cfb-921d-6fc95e619eb3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-29T11:12:34Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-shadow-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 7031a75f-c7bb-4cfb-921d-6fc95e619eb3 [e2e-llm-inference-service] resourceVersion: '56715' [e2e-llm-inference-service] uid: e4bd422b-1780-4895-95f7-efa35b6c399c [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-inference--ip-8bbb5aea.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:12:05Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7031a75f-c7bb-4cfb-921d-6fc95e619eb3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-29T11:12:05Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 7031a75f-c7bb-4cfb-921d-6fc95e619eb3 [e2e-llm-inference-service] resourceVersion: '56303' [e2e-llm-inference-service] uid: d55aa8a6-233d-4ed3-b2b3-345c39e49fb0 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:12:04Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7031a75f-c7bb-4cfb-921d-6fc95e619eb3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:extensionRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:portNumber: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPortNumber: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-29T11:12:04Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 7031a75f-c7bb-4cfb-921d-6fc95e619eb3 [e2e-llm-inference-service] resourceVersion: '56224' [e2e-llm-inference-service] uid: 04572ced-c316-452e-902c-93572619fc06 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] extensionRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-service [e2e-llm-inference-service] portNumber: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPortNumber: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parent: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '1970-01-01T00:00:00Z' [e2e-llm-inference-service] message: Waiting for controller [e2e-llm-inference-service] reason: Pending [e2e-llm-inference-service] status: Unknown [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Status [e2e-llm-inference-service] name: default [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67 [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:27:03Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 6d9fffb56c [e2e-llm-inference-service] timestamp: '2026-07-29T11:26:34Z' [e2e-llm-inference-service] window: 28.122s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 16899972n [e2e-llm-inference-service] memory: 24376Ki [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 87559953n [e2e-llm-inference-service] memory: 2401088Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schexzxnd [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:27:03Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 7fc69df7b8 [e2e-llm-inference-service] timestamp: '2026-07-29T11:26:42Z' [e2e-llm-inference-service] window: 15.572s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 56707937n [e2e-llm-inference-service] memory: 34792Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_inference_service] [2026-07-29T11:27:03.840071] end - ❌ 902.327s: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-07-29T11:12:35Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-29T11:12:35Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-29T11:14:04Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:12:35Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:12:06Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-29T11:12:35Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-29T11:12:41Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-29T11:12:41Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:12:35Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ---------------------------- Captured log teardown ----------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.conftest:conftest.py:168 Skipping deletion of namespace e2e-test-llm-inference-service-48639af5 (SKIP_DELETION_ON_FAILURE) [e2e-llm-inference-service] _ test_llm_autoscaling_hpa_lws[router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-hpa] _ [e2e-llm-inference-service] [gw1] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-lws', 'prometheus-scrape', 'scaling-hpa'], prompt='KSer... {'name': 'scaling-hpa-autoscale-hpa-lws-b344a3ff'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.autoscaling_hpa [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator-lws", [e2e-llm-inference-service] "prometheus-scrape", [e2e-llm-inference-service] "scaling-hpa", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="autoscale-hpa-lws", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_multi_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_autoscaling_hpa_lws(test_case: TestCase): [e2e-llm-inference-service] """HPA + LWS: HPA exists with WVA annotations; pods scale under load.""" [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] kserve_client = _new_kserve_client() [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] ns = test_case.namespace [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > _create_and_wait(kserve_client, test_case) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:670: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-lws', 'prometheus-scrape', 'scaling-hpa'], prompt='KSer... {'name': 'scaling-hpa-autoscale-hpa-lws-b344a3ff'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] def _create_and_wait(kserve_client, test_case): [e2e-llm-inference-service] """Create LLMISVC and wait for it to be ready.""" [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:482: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...ale-hpa-b29acdba'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-lws-b344a3ff'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-07-29T11:24:21.267977', start_time = 1785324261.2684317 [e2e-llm-inference-service] duration = 900.334662437439, timestamp_end = '2026-07-29T11:39:21.603096' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....autoscale-hpa-b29acdba'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-lws-b344a3ff'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1376: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7fecdc865260> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1387: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-29T11:25:22Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-29T11:24:53Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-29T11:24:37Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-29T11:25:22Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-29T11:25:22Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-29T11:24:53Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-29T11:25:22Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:24:53Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:24:53Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1371: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:82 Created test namespace e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret seaweedfs-s3-creds from kserve-ci-e2e-test to e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret storage-config from kserve-ci-e2e-test to e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:178 Patched default SA in e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 with secret seaweedfs-s3-creds [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:162 ConfigMap odh-kserve-custom-ca-bundle already exists in e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:159 Copied ConfigMap odh-trusted-ca-bundle from kserve-ci-e2e-test to e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig router-managed-autoscale-hpa-lw-1aa98714 in namespace e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig router-managed-autoscale-hpa-lw-1aa98714 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig router-managed-autoscale-hpa-lw-1aa98714 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig workload-llmd-simulator-lws-aut-fe7a55cc in namespace e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig workload-llmd-simulator-lws-aut-fe7a55cc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig workload-llmd-simulator-lws-aut-fe7a55cc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig prometheus-scrape-autoscale-hpa-b29acdba in namespace e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig prometheus-scrape-autoscale-hpa-b29acdba [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig prometheus-scrape-autoscale-hpa-b29acdba [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig scaling-hpa-autoscale-hpa-lws-b344a3ff in namespace e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig scaling-hpa-autoscale-hpa-lws-b344a3ff [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig scaling-hpa-autoscale-hpa-lws-b344a3ff [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_autoscaling_hpa_lws] [2026-07-29T11:24:20.900468] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-llmd-simulator-lws', 'prometheus-scrape', 'scaling-hpa'], prompt='KServe is a', service_name='autoscale-hpa-lws', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, namespace='e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2', before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-lws', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-lw-1aa98714'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-lws-aut-fe7a55cc'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-b29acdba'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-lws-b344a3ff'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-29T11:24:20.913228] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-lws', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-lw-1aa98714'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-lws-aut-fe7a55cc'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-b29acdba'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-lws-b344a3ff'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-29T11:24:21.267767] end - ✅ in 0.354s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-29T11:24:21.267977] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-lws', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-lw-1aa98714'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-lws-aut-fe7a55cc'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-b29acdba'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-lws-b344a3ff'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-29T11:24:37Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-29T11:24:37Z', 'message': 'Inference Pool e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-29T11:24:37Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-29T11:24:37Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-29T11:24:37Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-29T11:24:37Z', 'message': 'HPA conditions not yet available', 'reason': 'HPAProgressing', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-29T11:24:37Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:24:37Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:24:37Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-29T11:24:53Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-route: "False" (reason "BackendNotFound", message "backend(autoscale-hpa-lws-inference-pool-ip-3b720d6f.e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2.svc.cluster.local) not found")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-29T11:24:53Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-29T11:24:37Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-29T11:24:53Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-route: "False" (reason "BackendNotFound", message "backend(autoscale-hpa-lws-inference-pool-ip-3b720d6f.e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2.svc.cluster.local) not found")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-29T11:24:53Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-route: "False" (reason "BackendNotFound", message "backend(autoscale-hpa-lws-inference-pool-ip-3b720d6f.e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2.svc.cluster.local) not found")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-29T11:24:53Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-29T11:24:53Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:24:53Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:24:53Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-29T11:25:22Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-29T11:24:53Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-29T11:24:37Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-29T11:25:22Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-29T11:25:22Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-29T11:24:53Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-29T11:25:22Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:24:53Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:24:53Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1391 Timed out waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-29T11:25:22Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-29T11:24:53Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-29T11:24:37Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-29T11:25:22Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-29T11:25:22Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-29T11:24:53Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-29T11:25:22Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:24:53Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:24:53Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-07-29T11:39:21.603096] end - ❌ 900.335s: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-29T11:25:22Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-29T11:24:53Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-29T11:24:37Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-29T11:25:22Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-29T11:25:22Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-29T11:24:53Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-29T11:25:22Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:24:53Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:24:53Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [delete_llmisvc] [2026-07-29T11:39:21.603194] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-lws', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-lw-1aa98714'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-lws-aut-fe7a55cc'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-b29acdba'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-lws-b344a3ff'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 3 pod(s) for autoscale-hpa-lws still terminating: ['autoscale-hpa-lws-kserve-mn-0', 'autoscale-hpa-lws-kserve-mn-0-1', 'autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-4fh8h'] [e2e-llm-inference-service] assert not ['autoscale-hpa-lws-kserve-mn-0', 'autoscale-hpa-lws-kserve-mn-0-1', 'autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-4fh8h'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 1 pod(s) for autoscale-hpa-lws still terminating: ['autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-4fh8h'] [e2e-llm-inference-service] assert not ['autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-4fh8h'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [delete_llmisvc] [2026-07-29T11:40:11.979321] end - ✅ in 50.375s [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_autoscaling_hpa_lws] [2026-07-29T11:40:11.979477] end - ❌ 951.079s: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-29T11:25:22Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-29T11:24:53Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-29T11:24:37Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-29T11:25:22Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-29T11:25:22Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-29T11:24:53Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-29T11:25:22Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:24:53Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:24:53Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ---------------------------- Captured log teardown ----------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.conftest:conftest.py:168 Skipping deletion of namespace e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 (SKIP_DELETION_ON_FAILURE) [e2e-llm-inference-service] _ test_llm_inference_service[router-managed-workload-simulated-dp-ep-cpu-model-pvc] _ [e2e-llm-inference-service] [gw0] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-simulated-dp-ep-cpu', 'model-pvc'], prompt='KServe is a', service_name... {'name': 'model-pvc-llmisvc-model-pvc-rou-cfc8d654'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-gateway-ref", [e2e-llm-inference-service] "router-with-managed-route", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] expected_gateway="router-gateway-1", [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda tc: create_router_resources( [e2e-llm-inference-service] gateways=[ [e2e-llm-inference-service] make_router_gateway( [e2e-llm-inference-service] "router-gateway-1", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="custom-route-timeout-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="router-with-refs-test", [e2e-llm-inference-service] expected_gateway="router-gateway-1", [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda tc: create_router_resources( [e2e-llm-inference-service] gateways=[ [e2e-llm-inference-service] make_router_gateway( [e2e-llm-inference-service] "router-gateway-1", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] routes=[ [e2e-llm-inference-service] make_router_main_route( [e2e-llm-inference-service] "router-route-1", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] "router-gateway-1", [e2e-llm-inference-service] "router-with-refs-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] make_router_health_route( [e2e-llm-inference-service] "router-route-2", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] "router-gateway-1", [e2e-llm-inference-service] "router-with-refs-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=["router-managed", "workload-pd-cpu", "model-fb-opt-125m"], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="custom-route-timeout-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="router-with-refs-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] expected_gateway="router-gateway-2", [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda tc: create_router_resources( [e2e-llm-inference-service] gateways=[ [e2e-llm-inference-service] make_router_gateway( [e2e-llm-inference-service] "router-gateway-2", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] routes=[ [e2e-llm-inference-service] make_router_main_route( [e2e-llm-inference-service] "router-route-3", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] "router-gateway-2", [e2e-llm-inference-service] "router-with-refs-pd-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] make_router_health_route( [e2e-llm-inference-service] "router-route-4", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] "router-gateway-2", [e2e-llm-inference-service] "router-with-refs-pd-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-dp-ep-gpu", [e2e-llm-inference-service] "workload-dp-ep-prefill-gpu", [e2e-llm-inference-service] "model-deepseek-v2-lite", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="Delve into the multifaceted implications of a fully disaggregated cloud architecture, specifically " [e2e-llm-inference-service] "where the compute plane (P) and the data plane (D) are independently deployed and managed for a " [e2e-llm-inference-service] "geographically distributed, high-throughput, low-latency microservices ecosystem. Beyond the " [e2e-llm-inference-service] "fundamental challenges of network latency and data consistency, elaborate on the advanced " [e2e-llm-inference-service] "considerations and trade-offs inherent in such a setup: 1. Network Architecture and Protocols: " [e2e-llm-inference-service] "How would the network fabric and underlying protocols (e.g., RDMA, custom transport layers) need to " [e2e-llm-inference-service] "evolve to support optimal performance and minimize inter-plane communication overhead, especially for " [e2e-llm-inference-service] "synchronous operations? Discuss the role of network programmability (e.g., SDN, P4) in dynamically " [e2e-llm-inference-service] "optimizing routing and traffic flow between P and D. 2. Advanced Data Consistency and Durability: " [e2e-llm-inference-service] "Explore sophisticated data consistency models (e.g., causal consistency, strong eventual consistency) " [e2e-llm-inference-service] "and their applicability in balancing performance and data integrity across a globally distributed data plane. " [e2e-llm-inference-service] "Detail strategies for ensuring data durability and fault tolerance, including multi-region replication, " [e2e-llm-inference-service] "intelligent partitioning, and recovery mechanisms in the event of partial or full plane failures. " [e2e-llm-inference-service] "3. Dynamic Resource Orchestration and Cost Optimization: Analyze how an orchestration layer would intelligently " [e2e-llm-inference-service] "manage the independent scaling of compute (P) and data (D) resources, considering fluctuating workloads, " [e2e-llm-inference-service] "cost efficiency, and performance targets (e.g., using predictive analytics for resource provisioning). " [e2e-llm-inference-service] "Discuss mechanisms for dynamically reallocating compute nodes to different data partitions based on " [e2e-llm-inference-service] "workload patterns and data locality, potentially involving live migration strategies. " [e2e-llm-inference-service] "4. Security and Compliance in a Distributed Landscape: Address the enhanced security perimeter " [e2e-llm-inference-service] "challenges, including securing communication channels between P and D (encryption in transit, mutual TLS), " [e2e-llm-inference-service] "fine-grained access control to data at rest and in motion, and identity management across disaggregated " [e2e-llm-inference-service] "components. Discuss how such an architecture impacts compliance with regulatory frameworks (e.g., GDPR, HIPAA) " [e2e-llm-inference-service] "concerning data sovereignty, privacy, and auditability. 5. Operational Complexity and Observability: " [e2e-llm-inference-service] "Examine the increased complexity in monitoring, logging, and tracing across highly decoupled compute and " [e2e-llm-inference-service] "data planes. What specialized tooling and practices (e.g., distributed tracing with OpenTelemetry, advanced AIOps) " [e2e-llm-inference-service] "would be essential? How would incident response and troubleshooting differ in this disaggregated environment " [e2e-llm-inference-service] "compared to traditional integrated systems? Consider the challenges of pinpointing root causes across " [e2e-llm-inference-service] "independent failures. 6. Real-world Applicability and Future Trends: Identify specific industries " [e2e-llm-inference-service] "or use cases (e.g., high-frequency trading, IoT edge processing, large language model inference) " [e2e-llm-inference-service] "where the benefits of P/D disaggregation would strongly outweigh its complexities. " [e2e-llm-inference-service] "Conclude by speculating on emerging technologies or paradigms (e.g., serverless compute functions " [e2e-llm-inference-service] "directly interacting with object storage, in-memory disaggregation) that could further drive or " [e2e-llm-inference-service] "transform P/D disaggregation in cloud computing.", [e2e-llm-inference-service] max_tokens=2000, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_gpu, [e2e-llm-inference-service] pytest.mark.cluster_nvidia, [e2e-llm-inference-service] pytest.mark.cluster_nvidia_roce, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-no-scheduler", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.no_scheduler, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="This test simulates DP+EP that can run on CPU, the idea is to test the LWS-based deployment, " [e2e-llm-inference-service] "but without the resources requirements for DP+EP (GPUs and ROCe/IB).", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_multi_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler config tests [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-inline-config-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Chat completions endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] model_name="Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-configmap-ref", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-configmap-ref-test", [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda tc: create_scheduler_configmap(namespace=tc.namespace) [e2e-llm-inference-service] ], [e2e-llm-inference-service] after_test=[ [e2e-llm-inference-service] lambda tc: delete_scheduler_configmap(namespace=tc.namespace) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-replicas", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-ha-replicas-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-custom-template", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-custom-template-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler v0.6 → v0.7 migration tests. [e2e-llm-inference-service] # Deploy v0.6-style configs and verify the controller migrates them [e2e-llm-inference-service] # so the v0.7 scheduler boots successfully. [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-v06-pd-config-migration", [e2e-llm-inference-service] "workload-llmd-simulator-pd", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-v06-pd-migration-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-v06-nonzero-threshold-migration", [e2e-llm-inference-service] "workload-llmd-simulator-pd", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-v06-threshold-migration-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Standalone tokenizer — clean path: token-producer in inline config [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-tokenizer-kvcache", [e2e-llm-inference-service] "workload-llmd-simulator-kvcache", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="tokenizer-clean-path-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Standalone tokenizer — migration path: legacy precise-prefix-cache-scorer [e2e-llm-inference-service] # triggers auto-provisioned tokenizer without explicit tokenizer:{} field [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-precise-prefix-cache-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator-kvcache", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="tokenizer-migration-path-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Models endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="data"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/chat/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — LoRA adapter [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] model_name="publishers/{namespace}/models/lora-adapter-1", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "publishers/{namespace}/models/lora-adapter-1" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/lora-adapter-1", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/models (base + LoRA) [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=assert_models_contains( [e2e-llm-inference-service] "facebook/opt-125m", [e2e-llm-inference-service] "publishers/{namespace}/models/facebook/opt-125m", [e2e-llm-inference-service] "lora-adapter-1", [e2e-llm-inference-service] "publishers/{namespace}/models/lora-adapter-1", [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # PVC storage tests -- validate direct PVC volume mount with real vLLM serving [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] before_test=[lambda tc: ensure_pvc_with_model(namespace=tc.namespace)], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] before_test=[lambda tc: ensure_pvc_with_model(namespace=tc.namespace)], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] before_test=[lambda tc: ensure_pvc_with_model(namespace=tc.namespace)], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_multi_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_inference_service(test_case: TestCase): # noqa: F811 [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = KServeClient( [e2e-llm-inference-service] config_file=os.environ.get("KUBECONFIG", "~/.kube/config"), [e2e-llm-inference-service] client_configuration=client.Configuration(), [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] prefix = test_case.log_prefix [e2e-llm-inference-service] [e2e-llm-inference-service] test_failed = False [e2e-llm-inference-service] try: [e2e-llm-inference-service] print(f"{prefix} Creating LLMInferenceService {service_name}") [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] print(f"{prefix} Waiting for LLMInferenceService {service_name} to be ready") [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:866: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...pu-ll-699c687c'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-cfc8d654'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-07-29T11:27:25.566718', start_time = 1785324445.566987 [e2e-llm-inference-service] duration = 900.4009475708008, timestamp_end = '2026-07-29T11:42:25.967937' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....p-ep-cpu-ll-699c687c'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-cfc8d654'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1376: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7f21c95e3100> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1387: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-07-29T11:28:44Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-29T11:28:44Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-29T11:28:08Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-29T11:28:44Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-29T11:28:44Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-29T11:28:44Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:28:08Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:28:08Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1371: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:82 Created test namespace e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret seaweedfs-s3-creds from kserve-ci-e2e-test to e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret storage-config from kserve-ci-e2e-test to e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:178 Patched default SA in e2e-test-llm-inference-service-7c3633d1 with secret seaweedfs-s3-creds [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:162 ConfigMap odh-kserve-custom-ca-bundle already exists in e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:159 Copied ConfigMap odh-trusted-ca-bundle from kserve-ci-e2e-test to e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1959 Created PVC e2e-pvc-model-storage in namespace e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2099 Created model download Job e2e-pvc-model-download in namespace e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2124 Model download Job e2e-pvc-model-download completed successfully [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig router-managed-llmisvc-model-pv-2f105a64 in namespace e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig router-managed-llmisvc-model-pv-2f105a64 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig router-managed-llmisvc-model-pv-2f105a64 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig workload-simulated-dp-ep-cpu-ll-699c687c in namespace e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig workload-simulated-dp-ep-cpu-ll-699c687c [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig workload-simulated-dp-ep-cpu-ll-699c687c [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig model-pvc-llmisvc-model-pvc-rou-cfc8d654 in namespace e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig model-pvc-llmisvc-model-pvc-rou-cfc8d654 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig model-pvc-llmisvc-model-pvc-rou-cfc8d654 [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_inference_service] [2026-07-29T11:27:25.205384] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-simulated-dp-ep-cpu', 'model-pvc'], prompt='KServe is a', service_name='llmisvc-model-pvc-router-manage-2577e794', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, namespace='e2e-test-llm-inference-service-7c3633d1', before_test=[ at 0x7f21ca71fce0>], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-pvc-router-manage-2577e794', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-inference-service-7c3633d1', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-pv-2f105a64'}, [e2e-llm-inference-service] {'name': 'workload-simulated-dp-ep-cpu-ll-699c687c'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-cfc8d654'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-29T11:27:25.218231] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-pvc-router-manage-2577e794', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-inference-service-7c3633d1', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-pv-2f105a64'}, [e2e-llm-inference-service] {'name': 'workload-simulated-dp-ep-cpu-ll-699c687c'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-cfc8d654'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-29T11:27:25.566561] end - ✅ in 0.348s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-29T11:27:25.566718] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-pvc-router-manage-2577e794', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-inference-service-7c3633d1', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-pv-2f105a64'}, [e2e-llm-inference-service] {'name': 'workload-simulated-dp-ep-cpu-ll-699c687c'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-cfc8d654'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'RouterReady', 'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-07-29T11:28:08Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-29T11:28:08Z', 'message': 'Inference Pool e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-29T11:28:08Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-29T11:28:08Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-29T11:28:08Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-29T11:28:08Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:28:08Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:28:08Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-07-29T11:28:44Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-29T11:28:44Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-29T11:28:08Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-29T11:28:44Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-29T11:28:44Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-29T11:28:44Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:28:08Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:28:08Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 10: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-29T10:41:39Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 11: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-29T10:41:39Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 12: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-29T10:41:39Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1391 Timed out waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-07-29T11:28:44Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-29T11:28:44Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-29T11:28:08Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-29T11:28:44Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-29T11:28:44Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-29T11:28:44Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:28:08Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:28:08Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-07-29T11:42:25.967937] end - ❌ 900.401s: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-07-29T11:28:44Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-29T11:28:44Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-29T11:28:08Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-29T11:28:44Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-29T11:28:44Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-29T11:28:44Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:28:08Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:28:08Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:903 [router-managed-workload-simulated-dp-ep-cpu-model-pvc] ❌ ERROR: Failed to call llm inference service llmisvc-model-pvc-router-manage-2577e794: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-07-29T11:28:44Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-29T11:28:44Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-29T11:28:08Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-29T11:28:44Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-29T11:28:44Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-29T11:28:44Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:28:08Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:28:08Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:243 # Diagnostics for 'llmisvc-model-pvc-router-manage-2577e794' in 'e2e-test-llm-inference-service-7c3633d1' [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:244 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:247 # LLMInferenceService llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:256 apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] security.opendatahub.io/enable-auth: 'false' [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:27:25Z' [e2e-llm-inference-service] finalizers: [e2e-llm-inference-service] - serving.kserve.io/llmisvc-finalizer [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:security.opendatahub.io/enable-auth: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:baseRefs: {} [e2e-llm-inference-service] manager: OpenAPI-Generator [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-29T11:27:25Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:finalizers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] v:"serving.kserve.io/llmisvc-finalizer": {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-29T11:27:37Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:addresses: {} [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-router-route: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-scheduler: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-scheduler-latency-predictor: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-tokenizer: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-tracing: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-worker-data-parallel: {} [e2e-llm-inference-service] f:appliedConfigs: {} [e2e-llm-inference-service] f:conditions: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:router: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:gateways: {} [e2e-llm-inference-service] f:scheduler: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:inferencePool: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:service: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:url: {} [e2e-llm-inference-service] f:workloads: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:primary: {} [e2e-llm-inference-service] f:scheduler: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-29T11:28:44Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] resourceVersion: '67539' [e2e-llm-inference-service] uid: b2d8864a-033c-4334-a398-7bdaaa7f1714 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] baseRefs: [e2e-llm-inference-service] - name: router-managed-llmisvc-model-pv-2f105a64 [e2e-llm-inference-service] - name: workload-simulated-dp-ep-cpu-ll-699c687c [e2e-llm-inference-service] - name: model-pvc-llmisvc-model-pvc-rou-cfc8d654 [e2e-llm-inference-service] model: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uri: '' [e2e-llm-inference-service] status: [e2e-llm-inference-service] addresses: [e2e-llm-inference-service] - name: gateway-external-model-routing [e2e-llm-inference-service] url: http://a4d5e2ea28ab3498a9ca1aabff76d988-1031208529.us-east-1.elb.amazonaws.com/ [e2e-llm-inference-service] - name: gateway-external [e2e-llm-inference-service] url: http://a4d5e2ea28ab3498a9ca1aabff76d988-1031208529.us-east-1.elb.amazonaws.com/e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] - name: gateway-external [e2e-llm-inference-service] url: http://a4d5e2ea28ab3498a9ca1aabff76d988-1031208529.us-east-1.elb.amazonaws.com/publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m [e2e-llm-inference-service] - name: gateway-internal-model-routing [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/ [e2e-llm-inference-service] - name: gateway-internal [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] - name: gateway-internal [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-template: kserve-config-llm-decode-template [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-worker-data-parallel: kserve-config-llm-decode-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-template: kserve-config-llm-prefill-template [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-worker-data-parallel: kserve-config-llm-prefill-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-router-route: kserve-config-llm-router-route [e2e-llm-inference-service] serving.kserve.io/config-llm-scheduler: kserve-config-llm-scheduler [e2e-llm-inference-service] serving.kserve.io/config-llm-scheduler-latency-predictor: kserve-config-llm-scheduler-latency-predictor [e2e-llm-inference-service] serving.kserve.io/config-llm-template: kserve-config-llm-template [e2e-llm-inference-service] serving.kserve.io/config-llm-tokenizer: kserve-config-llm-tokenizer [e2e-llm-inference-service] serving.kserve.io/config-llm-tracing: kserve-config-llm-tracing [e2e-llm-inference-service] serving.kserve.io/config-llm-worker-data-parallel: kserve-config-llm-worker-data-parallel [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-29T11:28:44Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: HTTPRoutesReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-29T11:28:44Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: InferencePoolReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-29T11:28:08Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: PresetsCombined [e2e-llm-inference-service] - lastTransitionTime: '2026-07-29T11:28:44Z' [e2e-llm-inference-service] message: LWS is progressing [e2e-llm-inference-service] reason: Progressing [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: Ready [e2e-llm-inference-service] - lastTransitionTime: '2026-07-29T11:28:44Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: RouterReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-29T11:28:44Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: SchedulerWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-29T11:28:08Z' [e2e-llm-inference-service] message: LWS is progressing [e2e-llm-inference-service] reason: Progressing [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: WorkerWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-29T11:28:08Z' [e2e-llm-inference-service] message: LWS is progressing [e2e-llm-inference-service] reason: Progressing [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: WorkloadsReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] url: http://a4d5e2ea28ab3498a9ca1aabff76d988-1031208529.us-east-1.elb.amazonaws.com/e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:68 TIME NAMESPACE SOURCE TYPE REASON MESSAGE [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:69 -------------------------------------------------------------------------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None e2e-test-llm-inference-service-7c3633d1 Normal Scheduled Successfully assigned e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-zst94 to ip-10-0-140-240.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:27:09 e2e-test-llm-inference-service-7c3633d1 attachdetach-controller Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-b8b0e167-4bf3-4d8d-86ab-b583ee0ba857" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:27:11 e2e-test-llm-inference-service-7c3633d1 multus Normal AddedInterface Add eth0 [10.134.0.47/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:27:11 e2e-test-llm-inference-service-7c3633d1 kubelet/ip-10-0-140-240.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:13da3ee375cce7c9d4590e64ebb24ecf2e97ed4b2b64f9a4f2e3074ed4aafac1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:27:11 e2e-test-llm-inference-service-7c3633d1 kubelet/ip-10-0-140-240.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:27:11 e2e-test-llm-inference-service-7c3633d1 kubelet/ip-10-0-140-240.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:27:04 e2e-test-llm-inference-service-7c3633d1 job-controller Normal SuccessfulCreate Created pod: e2e-pvc-model-download-zst94 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:27:20 e2e-test-llm-inference-service-7c3633d1 job-controller Normal Completed Job completed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:27:04 e2e-test-llm-inference-service-7c3633d1 persistentvolume-controller Normal WaitForFirstConsumer waiting for first consumer to be created before binding [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:27:04 e2e-test-llm-inference-service-7c3633d1 persistentvolume-controller Normal ExternalProvisioning Waiting for a volume to be created either by the external provisioner 'ebs.csi.aws.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:27:04 e2e-test-llm-inference-service-7c3633d1 ebs.csi.aws.com_aws-ebs-csi-driver-controller-846c7568bf-vmfhs_43046ce3-27b6-434f-8292-da6b37871cde Normal Provisioning External provisioner is provisioning volume for claim "e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-storage" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:27:06 e2e-test-llm-inference-service-7c3633d1 ebs.csi.aws.com_aws-ebs-csi-driver-controller-846c7568bf-vmfhs_43046ce3-27b6-434f-8292-da6b37871cde Normal ProvisioningSucceeded Successfully provisioned volume pvc-b8b0e167-4bf3-4d8d-86ab-b583ee0ba857 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None e2e-test-llm-inference-service-7c3633d1 Normal Scheduled Successfully assigned e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1 to ip-10-0-129-80.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:28:09 e2e-test-llm-inference-service-7c3633d1 attachdetach-controller Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-b8b0e167-4bf3-4d8d-86ab-b583ee0ba857" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:28:10 e2e-test-llm-inference-service-7c3633d1 multus Normal AddedInterface Add eth0 [10.133.0.60/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:28:10 e2e-test-llm-inference-service-7c3633d1 kubelet/ip-10-0-129-80.ec2.internal Normal Pulling Pulling image "vllm/vllm-openai-cpu:v0.19.0" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:28:38 e2e-test-llm-inference-service-7c3633d1 kubelet/ip-10-0-129-80.ec2.internal Normal Pulled Successfully pulled image "vllm/vllm-openai-cpu:v0.19.0" in 28.585s (28.585s including waiting). Image size: 3531178100 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:28:38 e2e-test-llm-inference-service-7c3633d1 kubelet/ip-10-0-129-80.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:28:38 e2e-test-llm-inference-service-7c3633d1 kubelet/ip-10-0-129-80.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None e2e-test-llm-inference-service-7c3633d1 Normal Scheduled Successfully assigned e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 to ip-10-0-140-240.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:28:06 e2e-test-llm-inference-service-7c3633d1 statefulset-controller Normal SuccessfulCreate create Pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1 in StatefulSet llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 successful [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:28:09 e2e-test-llm-inference-service-7c3633d1 attachdetach-controller Warning FailedAttachVolume Multi-Attach error for volume "pvc-b8b0e167-4bf3-4d8d-86ab-b583ee0ba857" Volume is already used by pod(s) llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:28:06 e2e-test-llm-inference-service-7c3633d1 leaderworkerset Normal CreatingRevision Creating revision with key 7bc46f896d for a newly created LeaderWorkerSet [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:28:06 e2e-test-llm-inference-service-7c3633d1 leaderworkerset Normal GroupsProgressing Created leader statefulset llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:28:06 e2e-test-llm-inference-service-7c3633d1 leaderworkerset Normal GroupsProgressing Replicas are progressing, with 0 groups ready of total 1 groups [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:28:06 e2e-test-llm-inference-service-7c3633d1 statefulset-controller Normal SuccessfulCreate create Pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 in StatefulSet llmisvc-model-pvc-router-manage-2577e794-kserve-mn successful [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:28:06 e2e-test-llm-inference-service-7c3633d1 leaderworkerset Normal GroupsProgressing Created worker statefulset for leader pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:27:45 e2e-test-llm-inference-service-7c3633d1 OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-pvc-router-manage-2577e794-kserve-route-authn: failed to get HTTPRoute e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-pvc-router-manage-2577e794-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:28:06 e2e-test-llm-inference-service-7c3633d1 LLMInferenceServiceController Normal Created Created v1.Secret e2e-test-llm-inference-service-7c3633d1/llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:28:06 e2e-test-llm-inference-service-7c3633d1 LLMInferenceServiceController Normal Created Created v1.RoleBinding e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-scc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:28:06 e2e-test-llm-inference-service-7c3633d1 LLMInferenceServiceController Normal Created Created v1.ServiceAccount e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:28:06 e2e-test-llm-inference-service-7c3633d1 LLMInferenceServiceController Normal Created Created v1.LeaderWorkerSet e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:28:06 e2e-test-llm-inference-service-7c3633d1 LLMInferenceServiceController Normal Created Created v1.Service e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:28:06 e2e-test-llm-inference-service-7c3633d1 LLMInferenceServiceController Normal Created Created v1.ServiceAccount e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:28:06 e2e-test-llm-inference-service-7c3633d1 LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /e2e-test-llm-inference-service-5571cf2b7477d60b74b195d7a6a8f03e [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:28:06 e2e-test-llm-inference-service-7c3633d1 LLMInferenceServiceController Normal Created Created v1.Role e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:28:06 e2e-test-llm-inference-service-7c3633d1 LLMInferenceServiceController Normal Created Created v1.RoleBinding e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:28:44 e2e-test-llm-inference-service-7c3633d1 LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:38:25 e2e-test-llm-inference-service-7c3633d1 LLMInferenceServiceController Normal Updated Updated v1.Secret e2e-test-llm-inference-service-7c3633d1/llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:28:43 e2e-test-llm-inference-service-7c3633d1 LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:28:06 e2e-test-llm-inference-service-7c3633d1 replicaset-controller Normal SuccessfulCreate Created pod: llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schesf9dq [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:28:06 e2e-test-llm-inference-service-7c3633d1 deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheduler-66dcb4d65d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None e2e-test-llm-inference-service-7c3633d1 Normal Scheduled Successfully assigned e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schesf9dq to ip-10-0-133-86.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:28:07 e2e-test-llm-inference-service-7c3633d1 multus Normal AddedInterface Add eth0 [10.132.0.63/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:28:07 e2e-test-llm-inference-service-7c3633d1 kubelet/ip-10-0-133-86.ec2.internal Normal Pulled Container image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:28:07 e2e-test-llm-inference-service-7c3633d1 kubelet/ip-10-0-133-86.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 11:28:07 e2e-test-llm-inference-service-7c3633d1 kubelet/ip-10-0-133-86.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:172 ### Pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 (phase=Pending) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:212 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:228 # -- logs (current): unavailable ((400) [e2e-llm-inference-service] Reason: Bad Request [e2e-llm-inference-service] HTTP response headers: HTTPHeaderDict({'Audit-Id': '2a273de1-97bb-4977-9c92-f32fc9456ae7', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'Date': 'Wed, 29 Jul 2026 11:42:26 GMT', 'Content-Length': '236'}) [e2e-llm-inference-service] HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"container \"main\" in pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" is waiting to start: ContainerCreating","reason":"BadRequest","code":400} [e2e-llm-inference-service] [e2e-llm-inference-service] ) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:172 ### Pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1 (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:212 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:225 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:226 INFO 07-29 11:28:51 [importing.py:44] Triton is installed but 0 active driver(s) found (expected 1). Disabling Triton to prevent runtime errors. [e2e-llm-inference-service] INFO 07-29 11:28:51 [importing.py:68] Triton not installed or not compatible; certain GPU-related functions will not be available. [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 11:28:57 [utils.py:299] [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 11:28:57 [utils.py:299] █ █ █▄ ▄█ [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 11:28:57 [utils.py:299] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.19.0 [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 11:28:57 [utils.py:299] █▄█▀ █ █ █ █ model /mnt/models [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 11:28:57 [utils.py:299] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀ [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 11:28:57 [utils.py:299] [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 11:28:57 [utils.py:233] non-default args: {'model_tag': '/mnt/models', 'ssl_keyfile': '/var/run/kserve/tls/tls.key', 'ssl_certfile': '/var/run/kserve/tls/tls.crt', 'enable_ssl_refresh': True, 'model': '/mnt/models', 'served_model_name': ['facebook/opt-125m']} [e2e-llm-inference-service] (APIServer pid=1) WARNING 07-29 11:28:57 [arg_utils.py:1390] The global random seed is set to 0. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM. [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 11:29:14 [model.py:549] Resolved architecture: OPTForCausalLM [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 11:29:14 [model.py:1678] Using max model len 2048 [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 11:29:14 [vllm.py:790] Asynchronous scheduling is enabled. [e2e-llm-inference-service] INFO 07-29 11:29:26 [importing.py:44] Triton is installed but 0 active driver(s) found (expected 1). Disabling Triton to prevent runtime errors. [e2e-llm-inference-service] INFO 07-29 11:29:26 [importing.py:68] Triton not installed or not compatible; certain GPU-related functions will not be available. [e2e-llm-inference-service] (EngineCore pid=41) INFO 07-29 11:29:31 [core.py:105] Initializing a V1 LLM engine (v0.19.0) with config: model='/mnt/models', speculative_config=None, tokenizer='/mnt/models', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=2048, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cpu, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=facebook/opt-125m, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': , 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_images_per_batch': 0, 'compile_sizes': None, 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'dce': True, 'nan_asserts': False, 'epilogue_fusion': True, 'cpp.dynamic_threads': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': None, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} [e2e-llm-inference-service] (EngineCore pid=41) INFO 07-29 11:29:32 [cpu_worker.py:236] auto thread-binding list (id, physical core): [(4, 0), (5, 1), (6, 2), (7, 3)] [e2e-llm-inference-service] [W729 11:29:32.641364416 utils.cpp:76] Warning: numa_migrate_pages failed. errno: 1 (function init_cpu_threads_env) [e2e-llm-inference-service] [W729 11:29:32.641392857 utils.cpp:103] Warning: NUMA binding: Using MEMBIND policy for memory allocation on the NUMA nodes (0). Memory allocations will be strictly bound to these NUMA nodes. (function init_cpu_threads_env) [e2e-llm-inference-service] (EngineCore pid=41) INFO 07-29 11:29:32 [cpu_worker.py:109] OMP threads binding of Process 41: [e2e-llm-inference-service] (EngineCore pid=41) INFO 07-29 11:29:32 [cpu_worker.py:109] OMP tid: 41, core 4 [e2e-llm-inference-service] (EngineCore pid=41) INFO 07-29 11:29:32 [cpu_worker.py:109] OMP tid: 58, core 5 [e2e-llm-inference-service] (EngineCore pid=41) INFO 07-29 11:29:32 [cpu_worker.py:109] OMP tid: 59, core 6 [e2e-llm-inference-service] (EngineCore pid=41) INFO 07-29 11:29:32 [cpu_worker.py:109] OMP tid: 60, core 7 [e2e-llm-inference-service] (EngineCore pid=41) INFO 07-29 11:29:32 [cpu_worker.py:109] [e2e-llm-inference-service] (EngineCore pid=41) INFO 07-29 11:29:32 [parallel_state.py:1400] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.133.0.60:57143 backend=gloo [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] (EngineCore pid=41) INFO 07-29 11:29:32 [parallel_state.py:1716] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A [e2e-llm-inference-service] (EngineCore pid=41) INFO 07-29 11:29:32 [cpu_model_runner.py:71] Starting to load model /mnt/models... [e2e-llm-inference-service] (EngineCore pid=41) Loading pt checkpoint shards: 0% Completed | 0/1 [00:00}, FlowControl: {MaxBytes: unlimited, MaxRequests: unlimited, SaturationDetector: {PluginRef: utilization-detector}}, RequestHandler: {Parsers: [{PluginRef: openai-parser}, {PluginRef: anthropic-parser}, {PluginRef: vllmhttp-parser}]}}"} [e2e-llm-inference-service] {"level":"info","ts":1785324487.465368,"caller":"approximateprefix/plugin.go:88","msg":"Prefix DataProducer initialized","config":{"autoTune":true,"blockSizeTokens":16,"blockSize":0,"maxPrefixBlocksToMatch":2048,"maxPrefixTokensToMatch":131072,"lruCapacityPerServer":31250}} [e2e-llm-inference-service] {"level":"info","ts":1785324487.4654524,"caller":"approximateprefix/plugin.go:111","msg":"WARNING: configured blockSizeTokens is below the recommended minimum, overriding it.","blockSizeTokens":16,"minimum":64,"issue":"https://github.com/llm-d/llm-d-router/issues/1158"} [e2e-llm-inference-service] {"level":"info","ts":1785324487.465485,"caller":"datalayer/data_graph.go:116","msg":"auto-created default producer","producer":"approx-prefix-cache-producer/approx-prefix-cache-producer","dataKey":"PrefixCacheMatchInfoDataKey/approx-prefix-cache-producer","consumer":"no-hit-lru-scorer"} [e2e-llm-inference-service] {"level":"info","ts":1785324487.4655213,"caller":"datalayer/data_graph.go:116","msg":"auto-created default producer","producer":"token-producer/token-producer","dataKey":"TokenizedPrompt/token-producer","consumer":"approx-prefix-cache-producer"} [e2e-llm-inference-service] {"level":"info","ts":1785324487.4656408,"caller":"runner/runner.go:685","msg":"loaded configuration from file/text successfully"} [e2e-llm-inference-service] {"level":"info","ts":1785324487.4656537,"logger":"setup","caller":"runner/runner.go:308","msg":"EPP config after phase two","config":"{SchedulerConfig:{ProfileHandler: single-profile-handler/single-profile-handler, Profiles: map[default:{Filters: [], Scorers: [queue-scorer/queue-scorer: 2.000000, kv-cache-utilization-scorer/kv-cache-utilization-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000, no-hit-lru-scorer/no-hit-lru-scorer: 2.000000], Picker: max-score-picker/max-score-picker}]} SaturationDetector:0xc00080b8c0 DataConfig:{Sources:[{Plugin:0xc000986900 Extractors:[0xc00080bac0]}]} FlowControlConfig: ParserRegistry:0xc0009a8080}"} [e2e-llm-inference-service] {"level":"info","ts":1785324487.4850514,"logger":"setup","caller":"runner/runner.go:352","msg":"Setting pprof handlers"} [e2e-llm-inference-service] {"level":"info","ts":1785324487.4850821,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/mutex"} [e2e-llm-inference-service] {"level":"info","ts":1785324487.4851027,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/profile"} [e2e-llm-inference-service] {"level":"info","ts":1785324487.4851081,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/goroutine"} [e2e-llm-inference-service] {"level":"info","ts":1785324487.4851127,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/threadcreate"} [e2e-llm-inference-service] {"level":"info","ts":1785324487.4851167,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/"} [e2e-llm-inference-service] {"level":"info","ts":1785324487.4851213,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/cmdline"} [e2e-llm-inference-service] {"level":"info","ts":1785324487.4851253,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/symbol"} [e2e-llm-inference-service] {"level":"info","ts":1785324487.4851298,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/trace"} [e2e-llm-inference-service] {"level":"info","ts":1785324487.4851344,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/heap"} [e2e-llm-inference-service] {"level":"info","ts":1785324487.4851384,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/allocs"} [e2e-llm-inference-service] {"level":"info","ts":1785324487.4851425,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/block"} [e2e-llm-inference-service] {"level":"info","ts":1785324487.4851494,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/plugins/state"} [e2e-llm-inference-service] {"level":"info","ts":1785324487.4851544,"logger":"setup","caller":"runner/runner.go:373","msg":"parsed config","scheduler-config":"{ProfileHandler: single-profile-handler/single-profile-handler, Profiles: map[default:{Filters: [], Scorers: [queue-scorer/queue-scorer: 2.000000, kv-cache-utilization-scorer/kv-cache-utilization-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000, no-hit-lru-scorer/no-hit-lru-scorer: 2.000000], Picker: max-score-picker/max-score-picker}]}"} [e2e-llm-inference-service] {"level":"info","ts":1785324487.48518,"logger":"setup","caller":"datalayer/runtime.go:99","msg":"Configuring datalayer runtime","numSources":1} [e2e-llm-inference-service] {"level":"info","ts":1785324487.4851873,"logger":"setup","caller":"datalayer/runtime.go:118","msg":"Processing source","source":"metrics-data-source","numExtractors":1} [e2e-llm-inference-service] {"level":"info","ts":1785324487.485201,"logger":"setup","caller":"datalayer/runtime.go:147","msg":"Source configured","source":"metrics-data-source","extractors":["core-metrics-extractor/core-metrics-extractor"]} [e2e-llm-inference-service] {"level":"info","ts":1785324487.4852116,"logger":"setup","caller":"datalayer/runtime.go:206","msg":"Datalayer runtime configured","pollers":1,"notifiers":0,"endpointSources":0} [e2e-llm-inference-service] {"level":"info","ts":1785324487.4852436,"logger":"setup","caller":"runner/runner.go:833","msg":"Experimental Flow Control layer is disabled, using legacy admission control"} [e2e-llm-inference-service] {"level":"info","ts":1785324487.4853685,"logger":"setup","caller":"runner/runner.go:721","msg":"ExtProc server runner added to manager."} [e2e-llm-inference-service] {"level":"info","ts":1785324487.4853802,"logger":"setup","caller":"runner/runner.go:260","msg":"Controller manager starting"} [e2e-llm-inference-service] {"level":"info","ts":1785324487.4854062,"logger":"controller-runtime.metrics","caller":"server/server.go:208","msg":"Starting metrics server"} [e2e-llm-inference-service] {"level":"info","ts":1785324487.4857006,"caller":"runnable/grpc.go:35","msg":"gRPC server starting","name":"health"} [e2e-llm-inference-service] {"level":"info","ts":1785324487.4860897,"caller":"runnable/grpc.go:43","msg":"gRPC server listening","name":"health","port":9003} [e2e-llm-inference-service] {"level":"info","ts":1785324487.4864416,"logger":"controller-runtime.metrics","caller":"server/server.go:247","msg":"Serving metrics server","bindAddress":":9090","secure":false} [e2e-llm-inference-service] {"level":"info","ts":1785324487.4864595,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","source":"kind source: *v1.InferencePool"} [e2e-llm-inference-service] {"level":"info","ts":1785324487.4873877,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferenceobjective","controllerGroup":"llm-d.ai","controllerKind":"InferenceObjective","source":"kind source: *v1alpha2.InferenceObjective"} [e2e-llm-inference-service] {"level":"info","ts":1785324487.4876266,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferencemodelrewrite","controllerGroup":"llm-d.ai","controllerKind":"InferenceModelRewrite","source":"kind source: *v1alpha2.InferenceModelRewrite"} [e2e-llm-inference-service] {"level":"info","ts":1785324487.4886441,"caller":"runnable/grpc.go:35","msg":"gRPC server starting","name":"ext-proc"} [e2e-llm-inference-service] {"level":"info","ts":1785324487.4888217,"caller":"runnable/grpc.go:43","msg":"gRPC server listening","name":"ext-proc","port":9002} [e2e-llm-inference-service] {"level":"info","ts":1785324487.4889522,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"pod","controllerGroup":"","controllerKind":"Pod","source":"kind source: *v1.Pod"} [e2e-llm-inference-service] {"level":"info","ts":1785324487.4925687,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.InferenceObjective","reflector":"go/pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1785324487.4925792,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.InferenceModelRewrite","reflector":"go/pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1785324487.4926689,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.InferencePool","reflector":"go/pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1785324487.4936738,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Pod","reflector":"go/pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1785324487.5887778,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferencemodelrewrite","controllerGroup":"llm-d.ai","controllerKind":"InferenceModelRewrite"} [e2e-llm-inference-service] {"level":"info","ts":1785324487.5888026,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferencemodelrewrite","controllerGroup":"llm-d.ai","controllerKind":"InferenceModelRewrite","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1785324487.5898936,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferenceobjective","controllerGroup":"llm-d.ai","controllerKind":"InferenceObjective"} [e2e-llm-inference-service] {"level":"info","ts":1785324487.5899162,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferenceobjective","controllerGroup":"llm-d.ai","controllerKind":"InferenceObjective","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1785324487.688116,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool"} [e2e-llm-inference-service] {"level":"info","ts":1785324487.6881409,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1785324487.6882987,"caller":"controller/inferencepool_reconciler.go:46","msg":"Reconciling InferencePool","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"llmisvc-model-pvc-router-manage-2577e794-inference-pool","namespace":"e2e-test-llm-inference-service-7c3633d1"},"namespace":"e2e-test-llm-inference-service-7c3633d1","name":"llmisvc-model-pvc-router-manage-2577e794-inference-pool","reconcileID":"3d2ab449-4b63-40d7-97b3-1a57a8a3a1ae"} [e2e-llm-inference-service] {"level":"info","ts":1785324487.69025,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"pod","controllerGroup":"","controllerKind":"Pod"} [e2e-llm-inference-service] {"level":"info","ts":1785324487.6902719,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"pod","controllerGroup":"","controllerKind":"Pod","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1785324523.5850303,"caller":"controller/inferencepool_reconciler.go:46","msg":"Reconciling InferencePool","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"llmisvc-model-pvc-router-manage-2577e794-inference-pool","namespace":"e2e-test-llm-inference-service-7c3633d1"},"namespace":"e2e-test-llm-inference-service-7c3633d1","name":"llmisvc-model-pvc-router-manage-2577e794-inference-pool","reconcileID":"a7eaa04e-9bc4-4f28-971d-d0cefc99d73b"} [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-service [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] uid: 3dbe6d83-a419-4482-852d-d35b8bca6bca [e2e-llm-inference-service] resourceVersion: '67449' [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-29T11:28:38Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-29T11:28:38Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - ip: 10.132.0.63 [e2e-llm-inference-service] nodeName: ip-10-0-133-86.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] name: llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schesf9dq [e2e-llm-inference-service] uid: e9a55162-d774-4af1-9b73-1b1121a3eb6c [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] uid: 123b2038-0232-4a34-8073-17ce54dc67e1 [e2e-llm-inference-service] resourceVersion: '66912' [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-2577e794-kserve-mn- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] uid: 0c0efa00-4094-45d8-93e5-052c4139f31a [e2e-llm-inference-service] resourceVersion: '66937' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-leader [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] apps.kubernetes.io/pod-index: '0' [e2e-llm-inference-service] controller-revision-hash: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-5cf8fb667d [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/group-index: '0' [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/group-key: ae449cef3b581b6dbab6ee0440d253929da5cb58 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/template-revision-hash: 7bc46f896d [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/worker-index: '0' [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] statefulset.kubernetes.io/pod-name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.134.0.48/23"],"mac_address":"0a:58:0a:86:00:30","gateway_ips":["10.134.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.134.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.134.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.134.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.134.0.1"}],"ip_address":"10.134.0.48/23","gateway_ip":"10.134.0.1","role":"primary"}}' [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/size: '2' [e2e-llm-inference-service] openshift.io/scc: openshift-ai-llminferenceservice-scc [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: serviceaccount [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: StatefulSet [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] uid: 48386c49-81f3-451a-a70b-3d7a40581e32 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-140-240 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/size: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:apps.kubernetes.io/pod-index: {} [e2e-llm-inference-service] f:controller-revision-hash: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/name: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/template-revision-hash: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/worker-index: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:statefulset.kubernetes.io/pod-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"48386c49-81f3-451a-a70b-3d7a40581e32"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:add: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:hostname: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:subdomain: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 8Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] - name: kube-api-access-t72bv [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - vllm [e2e-llm-inference-service] - serve [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --served-model-name [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --enable-ssl-refresh [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: LWS_LEADER_ADDRESS [e2e-llm-inference-service] value: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0.llmisvc-model-pvc-router-manage-2577e794-kserve-mn.e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] - name: LWS_GROUP_SIZE [e2e-llm-inference-service] value: '2' [e2e-llm-inference-service] - name: LWS_WORKER_INDEX [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-t72bv [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] add: [e2e-llm-inference-service] - IPC_LOCK [e2e-llm-inference-service] - SYS_RAWIO [e2e-llm-inference-service] - NET_RAW [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000990000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] nodeName: ip-10-0-140-240.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c31,c30 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-5lcx4 [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-brhl9 [e2e-llm-inference-service] hostname: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 [e2e-llm-inference-service] subdomain: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Pending [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] reason: ContainersNotReady [e2e-llm-inference-service] message: 'containers with unready status: [main]' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] reason: ContainersNotReady [e2e-llm-inference-service] message: 'containers with unready status: [main]' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] hostIP: 10.0.140.240 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.140.240 [e2e-llm-inference-service] startTime: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] waiting: [e2e-llm-inference-service] reason: ContainerCreating [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: false [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] imageID: '' [e2e-llm-inference-service] started: false [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-t72bv [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1 [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] uid: 0b7f3375-271c-4793-ba1f-89b23a56fb52 [e2e-llm-inference-service] resourceVersion: '67463' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-worker [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] apps.kubernetes.io/pod-index: '1' [e2e-llm-inference-service] controller-revision-hash: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-74696b7c6f [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/group-index: '0' [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/group-key: ae449cef3b581b6dbab6ee0440d253929da5cb58 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/template-revision-hash: 7bc46f896d [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/worker-index: '1' [e2e-llm-inference-service] statefulset.kubernetes.io/pod-name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.133.0.60/23"],"mac_address":"0a:58:0a:85:00:3c","gateway_ips":["10.133.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.133.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.133.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.133.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.133.0.1"}],"ip_address":"10.133.0.60/23","gateway_ip":"10.133.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.133.0.60\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:85:00:3c\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/leader-name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/size: '2' [e2e-llm-inference-service] openshift.io/scc: openshift-ai-llminferenceservice-scc [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: serviceaccount [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: StatefulSet [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 [e2e-llm-inference-service] uid: 4b8f12be-9b4a-4ce8-81bf-de86577300e3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-129-80 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/leader-name: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/size: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:apps.kubernetes.io/pod-index: {} [e2e-llm-inference-service] f:controller-revision-hash: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/group-index: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/group-key: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/name: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/template-revision-hash: {} [e2e-llm-inference-service] f:statefulset.kubernetes.io/pod-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"4b8f12be-9b4a-4ce8-81bf-de86577300e3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:add: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:hostname: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:subdomain: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-29T11:28:10Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-29T11:28:39Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.133.0.60"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 8Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] - name: kube-api-access-z568m [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - vllm [e2e-llm-inference-service] - serve [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --served-model-name [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --enable-ssl-refresh [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: LWS_LEADER_ADDRESS [e2e-llm-inference-service] value: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0.llmisvc-model-pvc-router-manage-2577e794-kserve-mn.e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] - name: LWS_GROUP_SIZE [e2e-llm-inference-service] value: '2' [e2e-llm-inference-service] - name: LWS_WORKER_INDEX [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-z568m [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] add: [e2e-llm-inference-service] - IPC_LOCK [e2e-llm-inference-service] - SYS_RAWIO [e2e-llm-inference-service] - NET_RAW [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000990000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] nodeName: ip-10-0-129-80.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c31,c30 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-5lcx4 [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-brhl9 [e2e-llm-inference-service] hostname: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1 [e2e-llm-inference-service] subdomain: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-29T11:28:39Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-29T11:28:39Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-29T11:28:39Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] hostIP: 10.0.129.80 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.129.80 [e2e-llm-inference-service] podIP: 10.133.0.60 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.133.0.60 [e2e-llm-inference-service] startTime: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-07-29T11:28:38Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: docker.io/vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] imageID: docker.io/vllm/vllm-openai-cpu@sha256:a8257c201fd2f696c146615987be61579aa2052367b73c27bd3fb1ab8e90bd4b [e2e-llm-inference-service] containerID: cri-o://21be8d3388a3ccc86a6cd6eea666f608afb8836736640ad9fd5bb73a1cbf43a8 [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-z568m [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000990000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schesf9dq [e2e-llm-inference-service] generateName: llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheduler-66dcb4d65d- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] uid: e9a55162-d774-4af1-9b73-1b1121a3eb6c [e2e-llm-inference-service] resourceVersion: '67448' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 66dcb4d65d [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.10.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.132.0.63/23"],"mac_address":"0a:58:0a:84:00:3f","gateway_ips":["10.132.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.132.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.132.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.132.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.132.0.1"}],"ip_address":"10.132.0.63/23","gateway_ip":"10.132.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.132.0.63\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:84:00:3f\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheduler-66dcb4d65d [e2e-llm-inference-service] uid: bbc87784-5142-4c85-9819-3fab5f5ca557 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-133-86 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"bbc87784-5142-4c85-9819-3fab5f5ca557"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-29T11:28:07Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-29T11:28:38Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.132.0.63"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kube-api-access-bzh5j [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: llm-d.ai/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n- type:\ [e2e-llm-inference-service] \ single-profile-handler\n- type: queue-scorer\n- type: kv-cache-utilization-scorer\n\ [e2e-llm-inference-service] - type: prefix-cache-scorer\n- type: no-hit-lru-scorer\n- type: max-score-picker\n\ [e2e-llm-inference-service] - parameters:\n scheme: https\n type: metrics-data-source\nschedulingProfiles:\n\ [e2e-llm-inference-service] - name: default\n plugins:\n - pluginRef: queue-scorer\n weight: 2\n -\ [e2e-llm-inference-service] \ pluginRef: kv-cache-utilization-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n\ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: no-hit-lru-scorer\n weight: 2\n - pluginRef:\ [e2e-llm-inference-service] \ max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kube-api-access-bzh5j [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000990000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] nodeName: ip-10-0-133-86.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c31,c30 [e2e-llm-inference-service] fsGroup: 1000990000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-5lcx4 [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-2577e794-epp-sa-dockercfg-7958q [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-29T11:28:07Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-29T11:28:38Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-29T11:28:38Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] hostIP: 10.0.133.86 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.133.86 [e2e-llm-inference-service] podIP: 10.132.0.63 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.132.0.63 [e2e-llm-inference-service] startTime: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-07-29T11:28:07Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-llm-d-router-endpoint-picker@sha256:7fd1d61cb5505e026eebab0b9653758779c3dee241e37bea92a0bab961a48c50 [e2e-llm-inference-service] containerID: cri-o://2bc5422146ab7c6a535b045cd6de4b37b774e71117d99d3591d72df85c5987ee [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-bzh5j [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000990000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000990000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] uid: 3404683b-1eb4-488e-882c-214a34cc0769 [e2e-llm-inference-service] resourceVersion: '66917' [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] openshift.io/internal-registry-pull-secret-ref: llmisvc-model-pvc-router-manage-2577e794-epp-sa-dockercfg-7958q [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: b2d8864a-033c-4334-a398-7bdaaa7f1714 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: openshift.io/image-registry-pull-secrets_service-account-controller [e2e-llm-inference-service] operation: Apply [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:imagePullSecrets: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:openshift.io/internal-registry-pull-secret-ref: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] k:{"name":"llmisvc-model-pvc-router-manage-2577e794-epp-sa-dockercfg-7958q"}: {} [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b2d8864a-033c-4334-a398-7bdaaa7f1714"}: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"default-dockercfg-5lcx4"}: {} [e2e-llm-inference-service] k:{"name":"seaweedfs-s3-creds"}: {} [e2e-llm-inference-service] secrets: [e2e-llm-inference-service] - name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: default-dockercfg-5lcx4 [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-2577e794-epp-sa-dockercfg-7958q [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-5lcx4 [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-2577e794-epp-sa-dockercfg-7958q [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: ServiceAccount [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] uid: fd599f1a-1668-47c6-bbac-4baf53f3060b [e2e-llm-inference-service] resourceVersion: '66893' [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] openshift.io/internal-registry-pull-secret-ref: llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-brhl9 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: b2d8864a-033c-4334-a398-7bdaaa7f1714 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: openshift.io/image-registry-pull-secrets_service-account-controller [e2e-llm-inference-service] operation: Apply [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:imagePullSecrets: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:openshift.io/internal-registry-pull-secret-ref: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] k:{"name":"llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-brhl9"}: {} [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b2d8864a-033c-4334-a398-7bdaaa7f1714"}: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"default-dockercfg-5lcx4"}: {} [e2e-llm-inference-service] k:{"name":"seaweedfs-s3-creds"}: {} [e2e-llm-inference-service] secrets: [e2e-llm-inference-service] - name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: default-dockercfg-5lcx4 [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-brhl9 [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-5lcx4 [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-brhl9 [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: ServiceAccount [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-service [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] uid: 384f536d-849f-4d4b-aa96-1f0ab260457b [e2e-llm-inference-service] resourceVersion: '66959' [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: b2d8864a-033c-4334-a398-7bdaaa7f1714 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b2d8864a-033c-4334-a398-7bdaaa7f1714"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] targetPort: grpc [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] targetPort: grpc-health [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] targetPort: metrics [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] targetPort: zmq [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] clusterIP: 172.31.48.181 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.48.181 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] uid: a4441cf0-a013-4e7b-b8ce-cf080a9ebd45 [e2e-llm-inference-service] resourceVersion: '66908' [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: b2d8864a-033c-4334-a398-7bdaaa7f1714 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b2d8864a-033c-4334-a398-7bdaaa7f1714"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] targetPort: 8000 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] clusterIP: 172.31.175.30 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.175.30 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-74696b7c6f [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] uid: 4411289e-0fea-4faa-84f6-0f28095bd772 [e2e-llm-inference-service] resourceVersion: '66932' [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-worker [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] controller.kubernetes.io/hash: 74696b7c6f [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/group-index: '0' [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/group-key: ae449cef3b581b6dbab6ee0440d253929da5cb58 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/template-revision-hash: 7bc46f896d [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: StatefulSet [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 [e2e-llm-inference-service] uid: 4b8f12be-9b4a-4ce8-81bf-de86577300e3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:data: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:controller.kubernetes.io/hash: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/group-index: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/group-key: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/name: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/template-revision-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"4b8f12be-9b4a-4ce8-81bf-de86577300e3"}: {} [e2e-llm-inference-service] f:revision: {} [e2e-llm-inference-service] data: [e2e-llm-inference-service] spec: [e2e-llm-inference-service] template: [e2e-llm-inference-service] $patch: replace [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/leader-name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/size: '2' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-worker [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/group-index: '0' [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/group-key: ae449cef3b581b6dbab6ee0440d253929da5cb58 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/template-revision-hash: 7bc46f896d [e2e-llm-inference-service] spec: [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - args: [e2e-llm-inference-service] - --served-model-name [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --enable-ssl-refresh [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] command: [e2e-llm-inference-service] - vllm [e2e-llm-inference-service] - serve [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] name: main [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] add: [e2e-llm-inference-service] - IPC_LOCK [e2e-llm-inference-service] - SYS_RAWIO [e2e-llm-inference-service] - NET_RAW [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - mountPath: /home [e2e-llm-inference-service] name: home [e2e-llm-inference-service] - mountPath: /tmp [e2e-llm-inference-service] name: tmp-dir [e2e-llm-inference-service] - mountPath: /dev/shm [e2e-llm-inference-service] name: dshm [e2e-llm-inference-service] - mountPath: /models [e2e-llm-inference-service] name: model-cache [e2e-llm-inference-service] - mountPath: /var/run/kserve/tls [e2e-llm-inference-service] name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] - mountPath: /mnt/models [e2e-llm-inference-service] name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: home [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: tmp-dir [e2e-llm-inference-service] - emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 8Gi [e2e-llm-inference-service] name: dshm [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: model-cache [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] secretName: llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] revision: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ControllerRevision [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-5cf8fb667d [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] uid: 2295b421-f325-4544-a442-d300a7f07857 [e2e-llm-inference-service] resourceVersion: '66904' [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-leader [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] controller.kubernetes.io/hash: 5cf8fb667d [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/template-revision-hash: 7bc46f896d [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/worker-index: '0' [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/replicas: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: StatefulSet [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] uid: 48386c49-81f3-451a-a70b-3d7a40581e32 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:data: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/replicas: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:controller.kubernetes.io/hash: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/name: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/template-revision-hash: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/worker-index: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"48386c49-81f3-451a-a70b-3d7a40581e32"}: {} [e2e-llm-inference-service] f:revision: {} [e2e-llm-inference-service] data: [e2e-llm-inference-service] spec: [e2e-llm-inference-service] template: [e2e-llm-inference-service] $patch: replace [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/size: '2' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-leader [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/template-revision-hash: 7bc46f896d [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/worker-index: '0' [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] spec: [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - args: [e2e-llm-inference-service] - --served-model-name [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --enable-ssl-refresh [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] command: [e2e-llm-inference-service] - vllm [e2e-llm-inference-service] - serve [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] name: main [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] add: [e2e-llm-inference-service] - IPC_LOCK [e2e-llm-inference-service] - SYS_RAWIO [e2e-llm-inference-service] - NET_RAW [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - mountPath: /home [e2e-llm-inference-service] name: home [e2e-llm-inference-service] - mountPath: /tmp [e2e-llm-inference-service] name: tmp-dir [e2e-llm-inference-service] - mountPath: /dev/shm [e2e-llm-inference-service] name: dshm [e2e-llm-inference-service] - mountPath: /models [e2e-llm-inference-service] name: model-cache [e2e-llm-inference-service] - mountPath: /var/run/kserve/tls [e2e-llm-inference-service] name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] - mountPath: /mnt/models [e2e-llm-inference-service] name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: home [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: tmp-dir [e2e-llm-inference-service] - emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 8Gi [e2e-llm-inference-service] name: dshm [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: model-cache [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] secretName: llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] revision: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ControllerRevision [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheduler [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] uid: 2f478911-590f-4dd0-9e02-774dc96d9736 [e2e-llm-inference-service] resourceVersion: '67452' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: b2d8864a-033c-4334-a398-7bdaaa7f1714 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b2d8864a-033c-4334-a398-7bdaaa7f1714"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-29T11:28:38Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.10.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: llm-d.ai/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n-\ [e2e-llm-inference-service] \ type: single-profile-handler\n- type: queue-scorer\n- type: kv-cache-utilization-scorer\n\ [e2e-llm-inference-service] - type: prefix-cache-scorer\n- type: no-hit-lru-scorer\n- type: max-score-picker\n\ [e2e-llm-inference-service] - parameters:\n scheme: https\n type: metrics-data-source\nschedulingProfiles:\n\ [e2e-llm-inference-service] - name: default\n plugins:\n - pluginRef: queue-scorer\n weight: 2\n\ [e2e-llm-inference-service] \ - pluginRef: kv-cache-utilization-scorer\n weight: 2\n - pluginRef:\ [e2e-llm-inference-service] \ prefix-cache-scorer\n weight: 3\n - pluginRef: no-hit-lru-scorer\n\ [e2e-llm-inference-service] \ weight: 2\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: Recreate [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-29T11:28:38Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-29T11:28:38Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-29T11:28:38Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheduler-66dcb4d65d" [e2e-llm-inference-service] has successfully progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheduler-66dcb4d65d [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] uid: bbc87784-5142-4c85-9819-3fab5f5ca557 [e2e-llm-inference-service] resourceVersion: '67451' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 66dcb4d65d [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheduler [e2e-llm-inference-service] uid: 2f478911-590f-4dd0-9e02-774dc96d9736 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"2f478911-590f-4dd0-9e02-774dc96d9736"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-29T11:28:38Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 66dcb4d65d [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 66dcb4d65d [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.10.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: llm-d.ai/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n-\ [e2e-llm-inference-service] \ type: single-profile-handler\n- type: queue-scorer\n- type: kv-cache-utilization-scorer\n\ [e2e-llm-inference-service] - type: prefix-cache-scorer\n- type: no-hit-lru-scorer\n- type: max-score-picker\n\ [e2e-llm-inference-service] - parameters:\n scheme: https\n type: metrics-data-source\nschedulingProfiles:\n\ [e2e-llm-inference-service] - name: default\n plugins:\n - pluginRef: queue-scorer\n weight: 2\n\ [e2e-llm-inference-service] \ - pluginRef: kv-cache-utilization-scorer\n weight: 2\n - pluginRef:\ [e2e-llm-inference-service] \ prefix-cache-scorer\n weight: 3\n - pluginRef: no-hit-lru-scorer\n\ [e2e-llm-inference-service] \ weight: 2\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-rb [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] uid: 960ceb05-9cc1-481f-8548-6cfd9333629b [e2e-llm-inference-service] resourceVersion: '66950' [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: b2d8864a-033c-4334-a398-7bdaaa7f1714 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b2d8864a-033c-4334-a398-7bdaaa7f1714"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] apiGroup: rbac.authorization.k8s.io [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-role [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-scc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] uid: 35039119-9061-421e-81a4-de7d10f3868b [e2e-llm-inference-service] resourceVersion: '66889' [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: b2d8864a-033c-4334-a398-7bdaaa7f1714 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b2d8864a-033c-4334-a398-7bdaaa7f1714"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-prefill [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] apiGroup: rbac.authorization.k8s.io [e2e-llm-inference-service] kind: ClusterRole [e2e-llm-inference-service] name: openshift-ai-llminferenceservice-scc [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-role [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] uid: ad8cdfec-7d07-4bc2-ad7c-c2e8fb9439f7 [e2e-llm-inference-service] resourceVersion: '66935' [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: b2d8864a-033c-4334-a398-7bdaaa7f1714 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b2d8864a-033c-4334-a398-7bdaaa7f1714"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - llm-d.ai [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] - create [e2e-llm-inference-service] - update [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - delete [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-service-xmlnf [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-2577e794-epp-service- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] uid: 4853319a-4360-48bd-8a59-1f0b76eea6db [e2e-llm-inference-service] resourceVersion: '67450' [e2e-llm-inference-service] generation: 3 [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: llmisvc-model-pvc-router-manage-2577e794-epp-service [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-29T11:28:38Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-service [e2e-llm-inference-service] uid: 384f536d-849f-4d4b-aa96-1f0ab260457b [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-29T11:28:38Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"384f536d-849f-4d4b-aa96-1f0ab260457b"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.132.0.63 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] name: llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schesf9dq [e2e-llm-inference-service] uid: e9a55162-d774-4af1-9b73-1b1121a3eb6c [e2e-llm-inference-service] nodeName: ip-10-0-133-86.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-sw9vsh [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] uid: 40d3f3c8-7b0b-4948-a323-cc1ac79ab33c [e2e-llm-inference-service] resourceVersion: '66913' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] uid: a4441cf0-a013-4e7b-b8ce-cf080a9ebd45 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"a4441cf0-a013-4e7b-b8ce-cf080a9ebd45"}: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: null [e2e-llm-inference-service] ports: null [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-rb [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] uid: 960ceb05-9cc1-481f-8548-6cfd9333629b [e2e-llm-inference-service] resourceVersion: '66950' [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: b2d8864a-033c-4334-a398-7bdaaa7f1714 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b2d8864a-033c-4334-a398-7bdaaa7f1714"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] userNames: [e2e-llm-inference-service] - system:serviceaccount:e2e-test-llm-inference-service-7c3633d1:llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] groupNames: null [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-role [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-scc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] uid: 35039119-9061-421e-81a4-de7d10f3868b [e2e-llm-inference-service] resourceVersion: '66889' [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: b2d8864a-033c-4334-a398-7bdaaa7f1714 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b2d8864a-033c-4334-a398-7bdaaa7f1714"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] userNames: [e2e-llm-inference-service] - system:serviceaccount:e2e-test-llm-inference-service-7c3633d1:llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] - system:serviceaccount:e2e-test-llm-inference-service-7c3633d1:llmisvc-model-pvc-router-manage-2577e794-kserve-mn-prefill [e2e-llm-inference-service] groupNames: null [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-prefill [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] name: openshift-ai-llminferenceservice-scc [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-role [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] uid: ad8cdfec-7d07-4bc2-ad7c-c2e8fb9439f7 [e2e-llm-inference-service] resourceVersion: '66935' [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: b2d8864a-033c-4334-a398-7bdaaa7f1714 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b2d8864a-033c-4334-a398-7bdaaa7f1714"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - llm-d.ai [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - create [e2e-llm-inference-service] - delete [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - update [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:28:07Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b2d8864a-033c-4334-a398-7bdaaa7f1714"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-29T11:28:43Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-29T11:28:43Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-29T11:28:44Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-route [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: b2d8864a-033c-4334-a398-7bdaaa7f1714 [e2e-llm-inference-service] resourceVersion: '67537' [e2e-llm-inference-service] uid: aa42a63d-5da2-43b8-a56e-ca6258297ae6 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-29T11:28:07Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-29T11:28:43Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-29T11:28:07Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:28:07Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b2d8864a-033c-4334-a398-7bdaaa7f1714"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-29T11:28:43Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-29T11:28:43Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-29T11:28:44Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-route [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: b2d8864a-033c-4334-a398-7bdaaa7f1714 [e2e-llm-inference-service] resourceVersion: '67537' [e2e-llm-inference-service] uid: aa42a63d-5da2-43b8-a56e-ca6258297ae6 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-29T11:28:07Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-29T11:28:43Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-29T11:28:07Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b2d8864a-033c-4334-a398-7bdaaa7f1714"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:endpointPickerRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:number: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:matchLabels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPorts: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-29T11:28:43Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: b2d8864a-033c-4334-a398-7bdaaa7f1714 [e2e-llm-inference-service] resourceVersion: '67518' [e2e-llm-inference-service] uid: b5bb17df-3177-4616-8f84-a821e852b31a [e2e-llm-inference-service] spec: [e2e-llm-inference-service] appProtocol: http [e2e-llm-inference-service] endpointPickerRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-service [e2e-llm-inference-service] port: [e2e-llm-inference-service] number: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPorts: [e2e-llm-inference-service] - number: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-29T11:28:43Z' [e2e-llm-inference-service] message: Referenced by an HTTPRoute accepted by the parentRef Gateway [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-29T11:28:43Z' [e2e-llm-inference-service] message: Referenced ExtensionRef resolved successfully [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: networking.istio.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] kind: AuthPolicy [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:28:08Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-policies [e2e-llm-inference-service] app.kubernetes.io/managed-by: odh-model-controller [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b2d8864a-033c-4334-a398-7bdaaa7f1714"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:rules: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:authentication: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:public: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:anonymous: {} [e2e-llm-inference-service] f:credentials: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:overrides: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fairness: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:response: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:success: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:headers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:x-gateway-inference-fairness-id: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:x-gateway-inference-objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:targetRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-29T11:28:08Z' [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Accepted"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Enforced"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-29T11:28:11Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-route-authn [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: b2d8864a-033c-4334-a398-7bdaaa7f1714 [e2e-llm-inference-service] resourceVersion: '67154' [e2e-llm-inference-service] uid: 9155c5a9-9391-416e-9f03-d4387628ae3b [e2e-llm-inference-service] spec: [e2e-llm-inference-service] rules: [e2e-llm-inference-service] authentication: [e2e-llm-inference-service] public: [e2e-llm-inference-service] anonymous: {} [e2e-llm-inference-service] credentials: {} [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] overrides: [e2e-llm-inference-service] fairness: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] objective: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] response: [e2e-llm-inference-service] success: [e2e-llm-inference-service] headers: [e2e-llm-inference-service] x-gateway-inference-fairness-id: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.fairness [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] x-gateway-inference-objective: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.objective [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-route [e2e-llm-inference-service] status: [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-29T11:28:10Z' [e2e-llm-inference-service] message: AuthPolicy has been accepted [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-29T11:28:11Z' [e2e-llm-inference-service] message: AuthPolicy has been successfully enforced [e2e-llm-inference-service] reason: Enforced [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Enforced [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: leaderworkerset.x-k8s.io/v1 [e2e-llm-inference-service] kind: LeaderWorkerSet [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-worker [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: leaderworkerset.x-k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:conditions: {} [e2e-llm-inference-service] f:hpaPodSelector: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] manager: lws [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] - apiVersion: leaderworkerset.x-k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b2d8864a-033c-4334-a398-7bdaaa7f1714"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:leaderWorkerTemplate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:leaderTemplate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:add: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:size: {} [e2e-llm-inference-service] f:workerTemplate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:add: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:rolloutStrategy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupPolicy: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: b2d8864a-033c-4334-a398-7bdaaa7f1714 [e2e-llm-inference-service] resourceVersion: '66947' [e2e-llm-inference-service] uid: c3b11f7e-15ba-4acf-92a0-8af0cb1b1601 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] leaderWorkerTemplate: [e2e-llm-inference-service] leaderTemplate: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-leader [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] spec: [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - args: [e2e-llm-inference-service] - --served-model-name [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --enable-ssl-refresh [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] command: [e2e-llm-inference-service] - vllm [e2e-llm-inference-service] - serve [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] name: main [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] add: [e2e-llm-inference-service] - IPC_LOCK [e2e-llm-inference-service] - SYS_RAWIO [e2e-llm-inference-service] - NET_RAW [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - mountPath: /home [e2e-llm-inference-service] name: home [e2e-llm-inference-service] - mountPath: /tmp [e2e-llm-inference-service] name: tmp-dir [e2e-llm-inference-service] - mountPath: /dev/shm [e2e-llm-inference-service] name: dshm [e2e-llm-inference-service] - mountPath: /models [e2e-llm-inference-service] name: model-cache [e2e-llm-inference-service] - mountPath: /var/run/kserve/tls [e2e-llm-inference-service] name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] - mountPath: /mnt/models [e2e-llm-inference-service] name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: home [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: tmp-dir [e2e-llm-inference-service] - emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 8Gi [e2e-llm-inference-service] name: dshm [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: model-cache [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] restartPolicy: RecreateGroupOnPodRestart [e2e-llm-inference-service] size: 2 [e2e-llm-inference-service] workerTemplate: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-worker [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] spec: [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - args: [e2e-llm-inference-service] - --served-model-name [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --enable-ssl-refresh [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] command: [e2e-llm-inference-service] - vllm [e2e-llm-inference-service] - serve [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] name: main [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] add: [e2e-llm-inference-service] - IPC_LOCK [e2e-llm-inference-service] - SYS_RAWIO [e2e-llm-inference-service] - NET_RAW [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - mountPath: /home [e2e-llm-inference-service] name: home [e2e-llm-inference-service] - mountPath: /tmp [e2e-llm-inference-service] name: tmp-dir [e2e-llm-inference-service] - mountPath: /dev/shm [e2e-llm-inference-service] name: dshm [e2e-llm-inference-service] - mountPath: /models [e2e-llm-inference-service] name: model-cache [e2e-llm-inference-service] - mountPath: /var/run/kserve/tls [e2e-llm-inference-service] name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] - mountPath: /mnt/models [e2e-llm-inference-service] name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: home [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: tmp-dir [e2e-llm-inference-service] - emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 8Gi [e2e-llm-inference-service] name: dshm [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: model-cache [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] networkConfig: [e2e-llm-inference-service] subdomainPolicy: Shared [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] rolloutStrategy: [e2e-llm-inference-service] rollingUpdateConfiguration: [e2e-llm-inference-service] maxSurge: 0 [e2e-llm-inference-service] maxUnavailable: 1 [e2e-llm-inference-service] partition: 0 [e2e-llm-inference-service] type: RollingUpdate [e2e-llm-inference-service] startupPolicy: LeaderCreated [e2e-llm-inference-service] status: [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] message: Replicas are progressing [e2e-llm-inference-service] reason: GroupsProgressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Progressing [e2e-llm-inference-service] hpaPodSelector: leaderworkerset.sigs.k8s.io/name=llmisvc-model-pvc-router-manage-2577e794-kserve-mn,leaderworkerset.sigs.k8s.io/worker-index=0 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:28:07Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b2d8864a-033c-4334-a398-7bdaaa7f1714"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-29T11:28:07Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-scheduler [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: b2d8864a-033c-4334-a398-7bdaaa7f1714 [e2e-llm-inference-service] resourceVersion: '66992' [e2e-llm-inference-service] uid: 4e79b868-79fa-40d2-8a1a-54e80af5160b [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-2577e794-epp-service.e2e-test-llm-inference-service-7c3633d1.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-2577e794-epp-service.e2e-test-llm-inference-service-7c3633d1.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:28:43Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b2d8864a-033c-4334-a398-7bdaaa7f1714"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-29T11:28:43Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-shadow-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: b2d8864a-033c-4334-a398-7bdaaa7f1714 [e2e-llm-inference-service] resourceVersion: '67524' [e2e-llm-inference-service] uid: 6a1f45a2-fc53-42da-86ef-03d78151ab79 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-2577e794-inference--ip-950f33a3.e2e-test-llm-inference-service-7c3633d1.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc.e2e-test-llm-inference-service-7c3633d1.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:28:07Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b2d8864a-033c-4334-a398-7bdaaa7f1714"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-29T11:28:07Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: b2d8864a-033c-4334-a398-7bdaaa7f1714 [e2e-llm-inference-service] resourceVersion: '67007' [e2e-llm-inference-service] uid: c9f506e3-2d63-4936-8182-3514a58cf577 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc.e2e-test-llm-inference-service-7c3633d1.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc.e2e-test-llm-inference-service-7c3633d1.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:28:07Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b2d8864a-033c-4334-a398-7bdaaa7f1714"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-29T11:28:07Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-scheduler [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: b2d8864a-033c-4334-a398-7bdaaa7f1714 [e2e-llm-inference-service] resourceVersion: '66992' [e2e-llm-inference-service] uid: 4e79b868-79fa-40d2-8a1a-54e80af5160b [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-2577e794-epp-service.e2e-test-llm-inference-service-7c3633d1.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-2577e794-epp-service.e2e-test-llm-inference-service-7c3633d1.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:28:43Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b2d8864a-033c-4334-a398-7bdaaa7f1714"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-29T11:28:43Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-shadow-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: b2d8864a-033c-4334-a398-7bdaaa7f1714 [e2e-llm-inference-service] resourceVersion: '67524' [e2e-llm-inference-service] uid: 6a1f45a2-fc53-42da-86ef-03d78151ab79 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-2577e794-inference--ip-950f33a3.e2e-test-llm-inference-service-7c3633d1.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc.e2e-test-llm-inference-service-7c3633d1.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:28:07Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b2d8864a-033c-4334-a398-7bdaaa7f1714"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-29T11:28:07Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: b2d8864a-033c-4334-a398-7bdaaa7f1714 [e2e-llm-inference-service] resourceVersion: '67007' [e2e-llm-inference-service] uid: c9f506e3-2d63-4936-8182-3514a58cf577 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc.e2e-test-llm-inference-service-7c3633d1.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc.e2e-test-llm-inference-service-7c3633d1.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:28:07Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b2d8864a-033c-4334-a398-7bdaaa7f1714"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-29T11:28:07Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-scheduler [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: b2d8864a-033c-4334-a398-7bdaaa7f1714 [e2e-llm-inference-service] resourceVersion: '66992' [e2e-llm-inference-service] uid: 4e79b868-79fa-40d2-8a1a-54e80af5160b [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-2577e794-epp-service.e2e-test-llm-inference-service-7c3633d1.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-2577e794-epp-service.e2e-test-llm-inference-service-7c3633d1.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:28:43Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b2d8864a-033c-4334-a398-7bdaaa7f1714"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-29T11:28:43Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-shadow-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: b2d8864a-033c-4334-a398-7bdaaa7f1714 [e2e-llm-inference-service] resourceVersion: '67524' [e2e-llm-inference-service] uid: 6a1f45a2-fc53-42da-86ef-03d78151ab79 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-2577e794-inference--ip-950f33a3.e2e-test-llm-inference-service-7c3633d1.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc.e2e-test-llm-inference-service-7c3633d1.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:28:07Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b2d8864a-033c-4334-a398-7bdaaa7f1714"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-29T11:28:07Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: b2d8864a-033c-4334-a398-7bdaaa7f1714 [e2e-llm-inference-service] resourceVersion: '67007' [e2e-llm-inference-service] uid: c9f506e3-2d63-4936-8182-3514a58cf577 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc.e2e-test-llm-inference-service-7c3633d1.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc.e2e-test-llm-inference-service-7c3633d1.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b2d8864a-033c-4334-a398-7bdaaa7f1714"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:extensionRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:portNumber: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPortNumber: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-29T11:28:06Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: b2d8864a-033c-4334-a398-7bdaaa7f1714 [e2e-llm-inference-service] resourceVersion: '66971' [e2e-llm-inference-service] uid: 202f848a-e769-426b-b408-65a9e65345c6 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] extensionRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-service [e2e-llm-inference-service] portNumber: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPortNumber: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parent: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '1970-01-01T00:00:00Z' [e2e-llm-inference-service] message: Waiting for controller [e2e-llm-inference-service] reason: Pending [e2e-llm-inference-service] status: Unknown [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Status [e2e-llm-inference-service] name: default [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1 [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:42:26Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-worker [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] apps.kubernetes.io/pod-index: '1' [e2e-llm-inference-service] controller-revision-hash: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-74696b7c6f [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/group-index: '0' [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/group-key: ae449cef3b581b6dbab6ee0440d253929da5cb58 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/template-revision-hash: 7bc46f896d [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/worker-index: '1' [e2e-llm-inference-service] statefulset.kubernetes.io/pod-name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1 [e2e-llm-inference-service] timestamp: '2026-07-29T11:42:13Z' [e2e-llm-inference-service] window: 14.532s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 4641962n [e2e-llm-inference-service] memory: 2414420Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schesf9dq [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] creationTimestamp: '2026-07-29T11:42:26Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 66dcb4d65d [e2e-llm-inference-service] timestamp: '2026-07-29T11:42:13Z' [e2e-llm-inference-service] window: 11.091s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 1262194n [e2e-llm-inference-service] memory: 22088Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_inference_service] [2026-07-29T11:42:26.980298] end - ❌ 901.774s: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-07-29T11:28:44Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-29T11:28:44Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-29T11:28:08Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-29T11:28:44Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-29T11:28:44Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-29T11:28:44Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:28:08Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:28:08Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ---------------------------- Captured log teardown ----------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.conftest:conftest.py:168 Skipping deletion of namespace e2e-test-llm-inference-service-7c3633d1 (SKIP_DELETION_ON_FAILURE) [e2e-llm-inference-service] _ test_llm_autoscaling_keda_lws[router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-keda] _ [e2e-llm-inference-service] [gw1] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-lws', 'prometheus-scrape', 'scaling-keda'], prompt='KSe... {'name': 'scaling-keda-autoscale-keda-lws-1337f511'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.autoscaling_keda [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator-lws", [e2e-llm-inference-service] "prometheus-scrape", [e2e-llm-inference-service] "scaling-keda", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="autoscale-keda-lws", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_multi_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_autoscaling_keda_lws(test_case: TestCase): [e2e-llm-inference-service] """KEDA + LWS: ScaledObject exists with WVA annotations; pods scale under load.""" [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] kserve_client = _new_kserve_client() [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] ns = test_case.namespace [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > _create_and_wait(kserve_client, test_case) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:728: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-lws', 'prometheus-scrape', 'scaling-keda'], prompt='KSe... {'name': 'scaling-keda-autoscale-keda-lws-1337f511'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] def _create_and_wait(kserve_client, test_case): [e2e-llm-inference-service] """Create LLMISVC and wait for it to be ready.""" [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:482: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...e-ked-231d315d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-lws-1337f511'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-07-29T11:40:13.266962', start_time = 1785325213.2675455 [e2e-llm-inference-service] duration = 900.5465221405029, timestamp_end = '2026-07-29T11:55:13.814070' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....toscale-ked-231d315d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-lws-1337f511'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1376: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7fecdc8668e0> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1387: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-29T11:41:38Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-29T11:41:38Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-29T11:40:35Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-29T11:41:38Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-29T11:41:38Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-29T11:41:38Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-29T11:41:38Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:41:38Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:41:38Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1371: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:82 Created test namespace e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret seaweedfs-s3-creds from kserve-ci-e2e-test to e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret storage-config from kserve-ci-e2e-test to e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:178 Patched default SA in e2e-test-llm-autoscaling-keda-lws-e541a132 with secret seaweedfs-s3-creds [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:162 ConfigMap odh-kserve-custom-ca-bundle already exists in e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:159 Copied ConfigMap odh-trusted-ca-bundle from kserve-ci-e2e-test to e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig router-managed-autoscale-keda-l-78828c4a in namespace e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig router-managed-autoscale-keda-l-78828c4a [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig router-managed-autoscale-keda-l-78828c4a [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig workload-llmd-simulator-lws-aut-1696d0b7 in namespace e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig workload-llmd-simulator-lws-aut-1696d0b7 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig workload-llmd-simulator-lws-aut-1696d0b7 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig prometheus-scrape-autoscale-ked-231d315d in namespace e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig prometheus-scrape-autoscale-ked-231d315d [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig prometheus-scrape-autoscale-ked-231d315d [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig scaling-keda-autoscale-keda-lws-1337f511 in namespace e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig scaling-keda-autoscale-keda-lws-1337f511 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig scaling-keda-autoscale-keda-lws-1337f511 [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_autoscaling_keda_lws] [2026-07-29T11:40:12.910133] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-llmd-simulator-lws', 'prometheus-scrape', 'scaling-keda'], prompt='KServe is a', service_name='autoscale-keda-lws', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, namespace='e2e-test-llm-autoscaling-keda-lws-e541a132', before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-keda-lws', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-keda-lws-e541a132', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-keda-l-78828c4a'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-lws-aut-1696d0b7'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-ked-231d315d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-lws-1337f511'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-29T11:40:12.922591] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-keda-lws', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-keda-lws-e541a132', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-keda-l-78828c4a'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-lws-aut-1696d0b7'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-ked-231d315d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-lws-1337f511'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-29T11:40:13.266779] end - ✅ in 0.344s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-29T11:40:13.266962] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-keda-lws', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-keda-lws-e541a132', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-keda-l-78828c4a'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-lws-aut-1696d0b7'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-ked-231d315d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-lws-1337f511'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-29T11:40:35Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-29T11:40:35Z', 'message': 'Inference Pool e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-29T11:40:35Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-29T11:40:35Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-29T11:40:35Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-29T11:40:35Z', 'message': 'ScaledObject not yet visible in cache', 'reason': 'ScaledObjectProgressing', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-29T11:40:35Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:40:35Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:40:35Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-29T11:41:38Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-29T11:41:38Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-29T11:40:35Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-29T11:41:38Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-29T11:41:38Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-29T11:41:38Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-29T11:41:38Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:41:38Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:41:38Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1391 Timed out waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-29T11:41:38Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-29T11:41:38Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-29T11:40:35Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-29T11:41:38Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-29T11:41:38Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-29T11:41:38Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-29T11:41:38Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:41:38Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:41:38Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-07-29T11:55:13.814070] end - ❌ 900.547s: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-29T11:41:38Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-29T11:41:38Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-29T11:40:35Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-29T11:41:38Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-29T11:41:38Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-29T11:41:38Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-29T11:41:38Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:41:38Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:41:38Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [delete_llmisvc] [2026-07-29T11:55:13.814162] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-keda-lws', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-keda-lws-e541a132', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-keda-l-78828c4a'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-lws-aut-1696d0b7'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-ked-231d315d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-lws-1337f511'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 3 pod(s) for autoscale-keda-lws still terminating: ['autoscale-keda-lws-kserve-mn-0', 'autoscale-keda-lws-kserve-mn-0-1', 'autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-4cljx'] [e2e-llm-inference-service] assert not ['autoscale-keda-lws-kserve-mn-0', 'autoscale-keda-lws-kserve-mn-0-1', 'autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-4cljx'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 1 pod(s) for autoscale-keda-lws still terminating: ['autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-4cljx'] [e2e-llm-inference-service] assert not ['autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-4cljx'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [delete_llmisvc] [2026-07-29T11:55:44.196662] end - ✅ in 30.382s [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_autoscaling_keda_lws] [2026-07-29T11:55:44.196793] end - ❌ 931.286s: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-29T11:41:38Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-29T11:41:38Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-29T11:40:35Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-29T11:41:38Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-29T11:41:38Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-29T11:41:38Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-29T11:41:38Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:41:38Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:41:38Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ---------------------------- Captured log teardown ----------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.conftest:conftest.py:168 Skipping deletion of namespace e2e-test-llm-autoscaling-keda-lws-e541a132 (SKIP_DELETION_ON_FAILURE) [e2e-llm-inference-service] _ test_llm_autoscaling_cleanup_hpa[router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-hpa] _ [e2e-llm-inference-service] [gw1] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-hpa'], prom... {'name': 'scaling-hpa-autoscale-cleanup-h-aa1ae037'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.autoscaling_hpa [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator-no-replicas", [e2e-llm-inference-service] "prometheus-scrape", [e2e-llm-inference-service] "scaling-hpa", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="autoscale-cleanup-hpa", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_autoscaling_cleanup_hpa(test_case: TestCase): [e2e-llm-inference-service] """Removing scaling config should delete HPA.""" [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] kserve_client = _new_kserve_client() [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] ns = test_case.namespace [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > _create_and_wait(kserve_client, test_case) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:892: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-hpa'], prom... {'name': 'scaling-hpa-autoscale-cleanup-h-aa1ae037'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] def _create_and_wait(kserve_client, test_case): [e2e-llm-inference-service] """Create LLMISVC and wait for it to be ready.""" [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:482: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...e-cle-5a67f5d1'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-cleanup-h-aa1ae037'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-07-29T11:55:45.167130', start_time = 1785326145.1676166 [e2e-llm-inference-service] duration = 900.8532440662384, timestamp_end = '2026-07-29T12:10:46.020863' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....toscale-cle-5a67f5d1'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-cleanup-h-aa1ae037'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1376: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7fecdc867560> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1387: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-29T11:56:32Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-29T11:56:32Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-29T11:56:32Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:55:58Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-29T11:56:32Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-29T11:56:32Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-29T11:56:32Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-29T11:56:32Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:56:32Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1371: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:82 Created test namespace e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret seaweedfs-s3-creds from kserve-ci-e2e-test to e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret storage-config from kserve-ci-e2e-test to e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:178 Patched default SA in e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40 with secret seaweedfs-s3-creds [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:162 ConfigMap odh-kserve-custom-ca-bundle already exists in e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:159 Copied ConfigMap odh-trusted-ca-bundle from kserve-ci-e2e-test to e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig router-managed-autoscale-cleanu-e5a6b97f in namespace e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig router-managed-autoscale-cleanu-e5a6b97f [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig router-managed-autoscale-cleanu-e5a6b97f [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig workload-llmd-simulator-no-repl-5d16e76d in namespace e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig workload-llmd-simulator-no-repl-5d16e76d [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig workload-llmd-simulator-no-repl-5d16e76d [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig prometheus-scrape-autoscale-cle-5a67f5d1 in namespace e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig prometheus-scrape-autoscale-cle-5a67f5d1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig prometheus-scrape-autoscale-cle-5a67f5d1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig scaling-hpa-autoscale-cleanup-h-aa1ae037 in namespace e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig scaling-hpa-autoscale-cleanup-h-aa1ae037 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig scaling-hpa-autoscale-cleanup-h-aa1ae037 [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_autoscaling_cleanup_hpa] [2026-07-29T11:55:44.904392] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-hpa'], prompt='KServe is a', service_name='autoscale-cleanup-hpa', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, namespace='e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40', before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-cleanup-hpa', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-cleanu-e5a6b97f'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-5d16e76d'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-cle-5a67f5d1'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-cleanup-h-aa1ae037'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-29T11:55:44.916919] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-cleanup-hpa', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-cleanu-e5a6b97f'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-5d16e76d'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-cle-5a67f5d1'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-cleanup-h-aa1ae037'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-29T11:55:45.166846] end - ✅ in 0.250s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-29T11:55:45.167130] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-cleanup-hpa', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-cleanu-e5a6b97f'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-5d16e76d'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-cle-5a67f5d1'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-cleanup-h-aa1ae037'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-29T11:55:58Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-29T11:55:58Z', 'message': 'Inference Pool e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-29T11:55:58Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:55:58Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-29T11:55:58Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-29T11:55:58Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-29T11:55:58Z', 'message': 'HPA conditions not yet available', 'reason': 'HPAProgressing', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-29T11:55:58Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:55:58Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-29T11:56:32Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-29T11:56:32Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-29T11:56:32Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:55:58Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-29T11:56:32Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-29T11:56:32Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-29T11:56:32Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-29T11:56:32Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:56:32Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1391 Timed out waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-29T11:56:32Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-29T11:56:32Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-29T11:56:32Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:55:58Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-29T11:56:32Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-29T11:56:32Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-29T11:56:32Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-29T11:56:32Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:56:32Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-07-29T12:10:46.020863] end - ❌ 900.853s: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-29T11:56:32Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-29T11:56:32Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-29T11:56:32Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:55:58Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-29T11:56:32Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-29T11:56:32Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-29T11:56:32Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-29T11:56:32Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:56:32Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [delete_llmisvc] [2026-07-29T12:10:46.020951] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-cleanup-hpa', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-cleanu-e5a6b97f'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-5d16e76d'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-cle-5a67f5d1'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-cleanup-h-aa1ae037'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 2 pod(s) for autoscale-cleanup-hpa still terminating: ['autoscale-cleanup-hpa-kserve-6cfdf55fc9-8dkpz', 'autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ddwlw'] [e2e-llm-inference-service] assert not ['autoscale-cleanup-hpa-kserve-6cfdf55fc9-8dkpz', 'autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ddwlw'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 1 pod(s) for autoscale-cleanup-hpa still terminating: ['autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ddwlw'] [e2e-llm-inference-service] assert not ['autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ddwlw'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [delete_llmisvc] [2026-07-29T12:11:56.321098] end - ✅ in 70.300s [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_autoscaling_cleanup_hpa] [2026-07-29T12:11:56.321232] end - ❌ 971.417s: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-29T11:56:32Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-29T11:56:32Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-29T11:56:32Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:55:58Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-29T11:56:32Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-29T11:56:32Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-29T11:56:32Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-29T11:56:32Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T11:56:32Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ---------------------------- Captured log teardown ----------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.conftest:conftest.py:168 Skipping deletion of namespace e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40 (SKIP_DELETION_ON_FAILURE) [e2e-llm-inference-service] =============================== warnings summary =============================== [e2e-llm-inference-service] llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-utilization-detector] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] /workspace/source/python/kserve/.venv/lib64/python3.11/site-packages/pytest_asyncio/plugin.py:761: DeprecationWarning: The event_loop fixture provided by pytest-asyncio has been redefined in [e2e-llm-inference-service] /workspace/source/test/e2e/conftest.py:43 [e2e-llm-inference-service] Replacing the event_loop fixture with a custom implementation is deprecated [e2e-llm-inference-service] and will lead to errors in the future. [e2e-llm-inference-service] If you want to request an asyncio event loop with a scope other than function [e2e-llm-inference-service] scope, use the "scope" argument to the asyncio mark when marking the tests. [e2e-llm-inference-service] If you want to return different types of event loops, use the event_loop_policy [e2e-llm-inference-service] fixture. [e2e-llm-inference-service] [e2e-llm-inference-service] warnings.warn( [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-utilization-detector] [e2e-llm-inference-service] llmisvc/test_flow_control.py:47: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-concurrency-detector] [e2e-llm-inference-service] llmisvc/test_flow_control.py:47: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-inline-config-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-qwen2.5-0.5b] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-configmap-ref-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-replicas-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-custom-template-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-nonzero-threshold-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-tokenizer-kvcache-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator2] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service_stop.py::test_llm_stop_feature[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service_stop.py:40: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_tls.py::test_llm_tls_resources[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_tls.py:92: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html [e2e-llm-inference-service] - generated xml file: /workspace/artifacts-dir/junit_e2e-llminferenceservice_and_cluster_cpu_and_not_autoscaling_and_not_tracing.xml - [e2e-llm-inference-service] --------------------------------- JSON report ---------------------------------- [e2e-llm-inference-service] report saved to: /workspace/artifacts-dir/e2e_results-llminferenceservice_and_cluster_cpu_and_not_autoscaling_and_not_tracing.json [e2e-llm-inference-service] =========================== short test summary info ============================ [e2e-llm-inference-service] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_hpa_deployment[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_keda_deployment[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-keda] [e2e-llm-inference-service] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-pvc] [e2e-llm-inference-service] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_hpa_lws[cluster_cpu-cluster_multi_node-router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-pvc] [e2e-llm-inference-service] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_keda_lws[cluster_cpu-cluster_multi_node-router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-keda] [e2e-llm-inference-service] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_cleanup_hpa[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 7 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! [e2e-llm-inference-service] !!!!!!!!!!!! xdist.dsession.Interrupted: stopping after 5 failures !!!!!!!!!!!!! [e2e-llm-inference-service] ====== 7 failed, 42 passed, 3 skipped, 25 warnings in 5433.72s (1:30:33) ======= [must-gather] [must-gather ] OUT 2026-07-29T12:11:58.153131799Z Using must-gather plug-in image: quay.io/modh/must-gather:rhoai-2.24 [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: 7e1b384b-0e13-46c5-87f6-afaf6fa56fd7 [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.26" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [must-gather] [must-gather ] OUT 2026-07-29T12:11:58.201836053Z namespace/openshift-must-gather-hkldr created [must-gather] [must-gather ] OUT 2026-07-29T12:11:58.206588966Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-vffqw created [must-gather] [must-gather ] OUT 2026-07-29T12:11:58.229689929Z pod for plug-in image quay.io/modh/must-gather:rhoai-2.24 created [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:05.216591381Z [disk usage checker] Started [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:05.220183051Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70 [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:05.496923512Z Error from server (NotFound): namespaces "redhat-ods-operator" not found [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:05.673012260Z Wrote inspect data to must-gather. [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:05.673043795Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:05.673043795Z namespaces "redhat-ods-operator" not found [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:05.675453644Z Error getting logs from redhat-ods-operator [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:05.851485948Z Wrote inspect data to must-gather. [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:05.851521092Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:05.851521092Z namespaces "redhat-ods-monitoring" not found [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:05.853700882Z Error getting logs from redhat-ods-monitoring [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:06.030624590Z Wrote inspect data to must-gather. [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:06.030655220Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:06.030655220Z namespaces "redhat-ods-applications" not found [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:06.032938750Z Error getting logs from redhat-ods-applications [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:06.208414178Z Wrote inspect data to must-gather. [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:06.208444472Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:06.208444472Z namespaces "rhods-notebooks" not found [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:06.210729363Z Error getting logs from rhods-notebooks [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:06.378844879Z Wrote inspect data to must-gather. [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:06.378876249Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:06.378876249Z namespaces "rhoai-model-registries" not found [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:06.380950269Z Error getting logs from rhoai-model-registries [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:06.553206330Z Wrote inspect data to must-gather. [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:06.553238100Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:06.553238100Z namespaces "istio-system" not found [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:06.555315392Z Error getting logs from istio-system [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:06.726887105Z Wrote inspect data to must-gather. [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:06.726917192Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:06.726917192Z namespaces "knative-serving" not found [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:06.729037460Z Error getting logs from knative-serving [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:06.901137547Z Wrote inspect data to must-gather. [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:06.901168764Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:06.901168764Z namespaces "redhat-ods-applications-auth-provider" not found [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:06.903495400Z Error getting logs from redhat-ods-applications-auth-provider [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:07.195407850Z Wrote inspect data to must-gather. [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:07.498937693Z Wrote inspect data to must-gather. [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:07.644067806Z error: the server doesn't have a resource type "auths" [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:07.731030532Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:07.733077688Z Error collecting info from [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:07.871913340Z error: the server doesn't have a resource type "monitorings" [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:07.955522559Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:07.957398379Z Error collecting info from [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:08.095154523Z error: the server doesn't have a resource type "featuretrackers" [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:08.181242684Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:08.183174572Z Error collecting info from [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:08.324235830Z error: the server doesn't have a resource type "codeflares" [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:08.409960151Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:08.412165971Z Error collecting info from [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:08.552484276Z error: the server doesn't have a resource type "dashboards" [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:08.642009768Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:08.644026501Z Error collecting info from [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:08.788060872Z error: the server doesn't have a resource type "datasciencepipelines" [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:08.876876321Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:08.879050113Z Error collecting info from [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:09.021735371Z error: the server doesn't have a resource type "feastoperators" [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:09.106515465Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:09.108703637Z Error collecting info from [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:09.248082173Z error: the server doesn't have a resource type "kserves" [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:09.336520071Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:09.338723785Z Error collecting info from [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:09.478848220Z error: the server doesn't have a resource type "kueues" [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:09.568487642Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:09.571055748Z Error collecting info from [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:09.718393452Z error: the server doesn't have a resource type "modelcontrollers" [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:09.808054073Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:09.810261736Z Error collecting info from [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:09.953540617Z error: the server doesn't have a resource type "modelmeshservings" [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:10.041704713Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:10.044851197Z Error collecting info from [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:10.189053342Z error: the server doesn't have a resource type "modelregistries" [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:10.224907491Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70 [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:10.283151189Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:10.285439051Z Error collecting info from [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:10.426820691Z error: the server doesn't have a resource type "rays" [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:10.514601270Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:10.516603097Z Error collecting info from [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:10.661310529Z error: the server doesn't have a resource type "trainingoperators" [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:10.752108524Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:10.754556422Z Error collecting info from [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:10.893997979Z error: the server doesn't have a resource type "trustyais" [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:10.980142793Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:10.982424881Z Error collecting info from [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:11.124042336Z error: the server doesn't have a resource type "workbenches" [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:11.213148148Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:11.215243289Z Error collecting info from [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:11.361370683Z error: the server doesn't have a resource type "hardwareprofiles" [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:11.451164175Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:11.453986741Z Error collecting info from [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:11.593211074Z error: the server doesn't have a resource type "llamastackoperators" [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:11.683171858Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:11.685498239Z Error collecting info from [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:12.434283769Z error: the server doesn't have a resource type "predictors" [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:12.568786139Z error: the server doesn't have a resource type "localmodelnodegroups" [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:12.709906642Z error: the server doesn't have a resource type "smcp" [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:12.848585462Z error: the server doesn't have a resource type "smm" [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:12.985591203Z error: the server doesn't have a resource type "smmr" [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:13.612817424Z error: the server doesn't have a resource type "knativeservings" [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:13.749294600Z error: the server doesn't have a resource type "configurations" [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:13.889840735Z error: the server doesn't have a resource type "routes" [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:14.037912838Z error: the server doesn't have a resource type "services" [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:14.176288949Z error: the server doesn't have a resource type "revisions" [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:14.774482245Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:14.916975552Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:15.005545147Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:15.084553877Z Wrote inspect data to must-gather. [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:15.229493672Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70 [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:15.254220533Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:15.395525521Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:15.490924065Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:15.571844731Z Wrote inspect data to must-gather. [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:15.750756262Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:15.892516100Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:15.970827154Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:16.043573207Z Wrote inspect data to must-gather. [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:16.217176249Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:16.354302486Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:16.440144260Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:16.513596501Z Wrote inspect data to must-gather. [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:16.688650783Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:16.826344896Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:16.903893929Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:16.977275086Z Wrote inspect data to must-gather. [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:17.150860502Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:17.292567826Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:17.377264701Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:17.462585344Z Wrote inspect data to must-gather. [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:17.644408731Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:17.789397129Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:17.872425708Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:17.953140759Z Wrote inspect data to must-gather. [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:18.127084987Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:18.266207693Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:18.342790878Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:18.416095440Z Wrote inspect data to must-gather. [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:18.589158770Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:18.732801190Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:18.813718801Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:18.891014270Z Wrote inspect data to must-gather. [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:19.066601229Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:19.203310443Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:19.280609037Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:19.355750858Z Wrote inspect data to must-gather. [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:19.525016196Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:19.661331860Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:19.737057196Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:19.809875382Z Wrote inspect data to must-gather. [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:19.991640756Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:20.128967265Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:20.203034628Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:20.234911041Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70 [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:20.278934415Z Wrote inspect data to must-gather. [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:20.451568808Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:20.589614606Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:20.666857015Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:20.741354339Z Wrote inspect data to must-gather. [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:20.913290606Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:21.050641877Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:21.133332516Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:21.205979219Z Wrote inspect data to must-gather. [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:21.381365389Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:22.122010490Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:22.279122304Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:22.385073544Z Wrote inspect data to must-gather. [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:22.557712115Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:23.318515314Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:23.479537323Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:23.593258980Z Wrote inspect data to must-gather. [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:23.764548888Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:24.504275813Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:24.661762850Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:24.775326937Z Wrote inspect data to must-gather. [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:24.951199230Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:25.239586100Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70 [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:25.759571271Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:25.934963787Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:26.041900083Z Wrote inspect data to must-gather. [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:26.212752059Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:27.006919186Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:27.187926702Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:27.308549592Z Wrote inspect data to must-gather. [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:27.487575784Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:28.508233338Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:28.696699434Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:28.848095645Z Wrote inspect data to must-gather. [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:29.024201389Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:29.744528528Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:29.931228934Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:30.055351467Z Wrote inspect data to must-gather. [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:30.229650281Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:30.244209670Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70 [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:30.983596777Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:31.162916904Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:31.287395924Z Wrote inspect data to must-gather. [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:31.458956036Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:32.194768951Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:32.378148058Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:32.498039976Z Wrote inspect data to must-gather. [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:32.675958886Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:33.432113811Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:33.617766688Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:33.741247259Z Wrote inspect data to must-gather. [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:33.914967226Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:34.654077665Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:34.835798130Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:34.955476934Z Wrote inspect data to must-gather. [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:35.129106211Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:35.249214089Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70 [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:35.872739864Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:36.055418805Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:36.177667519Z Wrote inspect data to must-gather. [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:36.348591701Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:37.094710093Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:37.281856579Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:37.401896284Z Wrote inspect data to must-gather. [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:37.575875530Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:38.320025901Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:38.502523827Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:38.627992766Z Wrote inspect data to must-gather. [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:38.802767590Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:39.570641796Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:39.756432351Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:39.875856845Z Wrote inspect data to must-gather. [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:40.052995375Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:40.254273817Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70 [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:40.793554064Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:40.973471308Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:41.094136962Z Wrote inspect data to must-gather. [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:41.263361721Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:41.990017224Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:42.167496411Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:42.295598746Z Wrote inspect data to must-gather. [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:42.467498408Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:43.204609508Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:43.387003191Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:43.511013195Z Wrote inspect data to must-gather. [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:43.685841792Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:44.431472838Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:44.611430079Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:44.733854088Z Wrote inspect data to must-gather. [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:44.906953283Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:45.259020150Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70 [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:45.667643954Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:45.854297659Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:45.982859982Z Wrote inspect data to must-gather. [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:46.160115208Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:46.891550481Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:47.078969109Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:47.201798278Z Wrote inspect data to must-gather. [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:47.387877639Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:48.144447241Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:48.336062193Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:48.461179905Z Wrote inspect data to must-gather. [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:48.634521188Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:49.385015575Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:49.571462081Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:49.697107309Z Wrote inspect data to must-gather. [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:49.871818616Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:50.263839180Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70 [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:50.602512059Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:50.807455340Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:50.929897338Z Wrote inspect data to must-gather. [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:51.102090423Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:51.856178557Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:52.036755932Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:52.155926800Z Wrote inspect data to must-gather. [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:52.330534264Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:53.079002929Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:53.266820214Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:53.390663371Z Wrote inspect data to must-gather. [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:53.563768839Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:54.289710990Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:54.468597984Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:54.589530165Z Wrote inspect data to must-gather. [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:54.766001630Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:55.268794895Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70 [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:55.498774971Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:55.682963562Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:55.805061814Z Wrote inspect data to must-gather. [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:55.978040047Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:56.729549353Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:56.911609539Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:57.037907879Z Wrote inspect data to must-gather. [must-gather] [must-gather-5w2ht] POD 2026-07-29T12:12:57.061496499Z Caches written to disk [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.072729526Z waiting for gather to complete [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.076092395Z downloading gather output [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.32584367Z receiving incremental file list [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.335833131Z ./ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.335933416Z aggregated-discovery-api.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.336276352Z aggregated-discovery-apis.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.337135242Z event-filter.html [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.33750886Z timestamp [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.337657147Z version [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.343361476Z cluster-scoped-resources/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.343376047Z cluster-scoped-resources/datasciencecluster.opendatahub.io/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.343382417Z cluster-scoped-resources/datasciencecluster.opendatahub.io/datascienceclusters/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.343414268Z cluster-scoped-resources/datasciencecluster.opendatahub.io/datascienceclusters/test-dsc.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.343516413Z cluster-scoped-resources/dscinitialization.opendatahub.io/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.343526434Z cluster-scoped-resources/dscinitialization.opendatahub.io/dscinitializations/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.343569216Z cluster-scoped-resources/dscinitialization.opendatahub.io/dscinitializations/test-dsci.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.343647649Z namespaces/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.3436581Z namespaces/kserve-ci-e2e-test/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.343713832Z namespaces/kserve-ci-e2e-test/kserve-ci-e2e-test.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.343779856Z namespaces/kserve-ci-e2e-test/apps.openshift.io/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.343821028Z namespaces/kserve-ci-e2e-test/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.343900211Z namespaces/kserve-ci-e2e-test/apps/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.343922722Z namespaces/kserve-ci-e2e-test/apps/daemonsets.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.344061119Z namespaces/kserve-ci-e2e-test/apps/deployments.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.344167944Z namespaces/kserve-ci-e2e-test/apps/replicasets.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.344265198Z namespaces/kserve-ci-e2e-test/apps/statefulsets.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.3443088Z namespaces/kserve-ci-e2e-test/autoscaling/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.344341912Z namespaces/kserve-ci-e2e-test/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.344396775Z namespaces/kserve-ci-e2e-test/batch/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.344437707Z namespaces/kserve-ci-e2e-test/batch/cronjobs.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.34451719Z namespaces/kserve-ci-e2e-test/batch/jobs.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.344550702Z namespaces/kserve-ci-e2e-test/build.openshift.io/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.344591744Z namespaces/kserve-ci-e2e-test/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.344675048Z namespaces/kserve-ci-e2e-test/build.openshift.io/builds.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.34472966Z namespaces/kserve-ci-e2e-test/core/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.344740051Z namespaces/kserve-ci-e2e-test/core/configmaps.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.344871097Z namespaces/kserve-ci-e2e-test/core/endpoints.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.3449383Z namespaces/kserve-ci-e2e-test/core/events.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.345020294Z namespaces/kserve-ci-e2e-test/core/persistentvolumeclaims.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.345128709Z namespaces/kserve-ci-e2e-test/core/pods.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.345196392Z namespaces/kserve-ci-e2e-test/core/replicationcontrollers.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.345324438Z namespaces/kserve-ci-e2e-test/core/secrets.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.345504997Z namespaces/kserve-ci-e2e-test/core/services.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.345556149Z namespaces/kserve-ci-e2e-test/discovery.k8s.io/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.34556905Z namespaces/kserve-ci-e2e-test/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.345634023Z namespaces/kserve-ci-e2e-test/image.openshift.io/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.345680605Z namespaces/kserve-ci-e2e-test/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.345712837Z namespaces/kserve-ci-e2e-test/k8s.ovn.org/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.345760209Z namespaces/kserve-ci-e2e-test/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.345848523Z namespaces/kserve-ci-e2e-test/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.345893895Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.345935867Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.345980939Z namespaces/kserve-ci-e2e-test/networking.k8s.io/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.346020521Z namespaces/kserve-ci-e2e-test/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.346090495Z namespaces/kserve-ci-e2e-test/policy/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.346134166Z namespaces/kserve-ci-e2e-test/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.346173298Z namespaces/kserve-ci-e2e-test/route.openshift.io/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.34620986Z namespaces/kserve-ci-e2e-test/route.openshift.io/routes.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.346247482Z namespaces/kuadrant-system/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.346296974Z namespaces/kuadrant-system/kuadrant-system.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.346365567Z namespaces/kuadrant-system/apps.openshift.io/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.346397659Z namespaces/kuadrant-system/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.346445091Z namespaces/kuadrant-system/apps/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.346489673Z namespaces/kuadrant-system/apps/daemonsets.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.346571987Z namespaces/kuadrant-system/apps/deployments.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.346893622Z namespaces/kuadrant-system/apps/replicasets.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.347237978Z namespaces/kuadrant-system/apps/statefulsets.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.34726003Z namespaces/kuadrant-system/autoscaling/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.347338363Z namespaces/kuadrant-system/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.347365615Z namespaces/kuadrant-system/batch/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.347424417Z namespaces/kuadrant-system/batch/cronjobs.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.347503571Z namespaces/kuadrant-system/batch/jobs.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.347543323Z namespaces/kuadrant-system/build.openshift.io/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.347574474Z namespaces/kuadrant-system/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.347666679Z namespaces/kuadrant-system/build.openshift.io/builds.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.34770514Z namespaces/kuadrant-system/core/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.347735162Z namespaces/kuadrant-system/core/configmaps.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.34791242Z namespaces/kuadrant-system/core/endpoints.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.348068157Z namespaces/kuadrant-system/core/events.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.348571051Z namespaces/kuadrant-system/core/persistentvolumeclaims.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.348646125Z namespaces/kuadrant-system/core/pods.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.349023803Z namespaces/kuadrant-system/core/replicationcontrollers.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.349196731Z namespaces/kuadrant-system/core/secrets.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.349433052Z namespaces/kuadrant-system/core/services.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.349563548Z namespaces/kuadrant-system/discovery.k8s.io/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.34959666Z namespaces/kuadrant-system/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.349727746Z namespaces/kuadrant-system/image.openshift.io/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.349742936Z namespaces/kuadrant-system/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.34980763Z namespaces/kuadrant-system/k8s.ovn.org/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.349835721Z namespaces/kuadrant-system/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.349926625Z namespaces/kuadrant-system/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.349973567Z namespaces/kuadrant-system/monitoring.coreos.com/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.35002081Z namespaces/kuadrant-system/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.350082492Z namespaces/kuadrant-system/networking.k8s.io/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.350117764Z namespaces/kuadrant-system/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.350170757Z namespaces/kuadrant-system/pods/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.350180987Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-5j7sb/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.350190018Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-5j7sb/authorino-7cf4765b85-5j7sb.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.350285852Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-5j7sb/authorino/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.350293402Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-5j7sb/authorino/authorino/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.350298013Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-5j7sb/authorino/authorino/logs/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.350329854Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-5j7sb/authorino/authorino/logs/current.log [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.365637745Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-5j7sb/authorino/authorino/logs/previous.insecure.log [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.365703999Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-5j7sb/authorino/authorino/logs/previous.log [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.36574567Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-rzd9v/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.365773502Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-rzd9v/authorino-operator-56fb6f6856-rzd9v.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.365879277Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-rzd9v/manager/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.365887547Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-rzd9v/manager/manager/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.365891837Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-rzd9v/manager/manager/logs/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.365919069Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-rzd9v/manager/manager/logs/current.log [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.366121558Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-rzd9v/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.366189541Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-rzd9v/manager/manager/logs/previous.log [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.366217993Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-9wzdw/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.366265345Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-9wzdw/dns-operator-controller-manager-ccbdf686-9wzdw.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.36636053Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-9wzdw/manager/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.36636863Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-9wzdw/manager/manager/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.36637278Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-9wzdw/manager/manager/logs/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.366407502Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-9wzdw/manager/manager/logs/current.log [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.366518107Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-9wzdw/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.36658897Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-9wzdw/manager/manager/logs/previous.log [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.366621802Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-gwjp6/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.366661714Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-gwjp6/kuadrant-console-plugin-598f86468b-gwjp6.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.366753098Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-gwjp6/kuadrant-console-plugin/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.366768079Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-gwjp6/kuadrant-console-plugin/kuadrant-console-plugin/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.366776259Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-gwjp6/kuadrant-console-plugin/kuadrant-console-plugin/logs/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.36678611Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-gwjp6/kuadrant-console-plugin/kuadrant-console-plugin/logs/current.log [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.366857043Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-gwjp6/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.insecure.log [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.366922196Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-gwjp6/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.log [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.366961428Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-m5dpg/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.3670007Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-m5dpg/kuadrant-operator-controller-manager-66866f5fcc-m5dpg.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.367162467Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-m5dpg/manager/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.367173138Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-m5dpg/manager/manager/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.367178848Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-m5dpg/manager/manager/logs/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.367189289Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-m5dpg/manager/manager/logs/current.log [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.384559527Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-m5dpg/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.38462714Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-m5dpg/manager/manager/logs/previous.log [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.384644071Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-c9rqr/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.384702354Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-c9rqr/limitador-limitador-85d9db4d5d-c9rqr.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.384796038Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-c9rqr/limitador/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.384805888Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-c9rqr/limitador/limitador/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.384811219Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-c9rqr/limitador/limitador/logs/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.38484228Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-c9rqr/limitador/limitador/logs/current.log [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.384928584Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-c9rqr/limitador/limitador/logs/previous.insecure.log [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.385028679Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-c9rqr/limitador/limitador/logs/previous.log [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.38505032Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-fdkhh/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.385106322Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-fdkhh/limitador-operator-controller-manager-774d94476f-fdkhh.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.385201457Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-fdkhh/manager/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.385216338Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-fdkhh/manager/manager/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.385220938Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-fdkhh/manager/manager/logs/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.385240959Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-fdkhh/manager/manager/logs/current.log [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.385386896Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-fdkhh/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.385455899Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-fdkhh/manager/manager/logs/previous.log [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.38548407Z namespaces/kuadrant-system/policy/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.385533773Z namespaces/kuadrant-system/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.385584735Z namespaces/kuadrant-system/route.openshift.io/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.385621897Z namespaces/kuadrant-system/route.openshift.io/routes.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.385672899Z namespaces/openshift-ingress/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.385723991Z namespaces/openshift-ingress/openshift-ingress.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.385791155Z namespaces/openshift-ingress/apps.openshift.io/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.385823906Z namespaces/openshift-ingress/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.385876939Z namespaces/openshift-ingress/apps/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.38591424Z namespaces/openshift-ingress/apps/daemonsets.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.385997944Z namespaces/openshift-ingress/apps/deployments.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.386264067Z namespaces/openshift-ingress/apps/replicasets.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.386456666Z namespaces/openshift-ingress/apps/statefulsets.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.386502998Z namespaces/openshift-ingress/autoscaling/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.38654828Z namespaces/openshift-ingress/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.386604283Z namespaces/openshift-ingress/batch/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.386634994Z namespaces/openshift-ingress/batch/cronjobs.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.386721638Z namespaces/openshift-ingress/batch/jobs.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.386767831Z namespaces/openshift-ingress/build.openshift.io/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.386811563Z namespaces/openshift-ingress/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.386895207Z namespaces/openshift-ingress/build.openshift.io/builds.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.386939569Z namespaces/openshift-ingress/core/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.386982161Z namespaces/openshift-ingress/core/configmaps.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.387420242Z namespaces/openshift-ingress/core/endpoints.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.387515466Z namespaces/openshift-ingress/core/events.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.387741677Z namespaces/openshift-ingress/core/persistentvolumeclaims.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.38782099Z namespaces/openshift-ingress/core/pods.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.388095443Z namespaces/openshift-ingress/core/replicationcontrollers.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.38822911Z namespaces/openshift-ingress/core/secrets.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.388432989Z namespaces/openshift-ingress/core/services.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.388518063Z namespaces/openshift-ingress/discovery.k8s.io/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.388563265Z namespaces/openshift-ingress/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.388638989Z namespaces/openshift-ingress/image.openshift.io/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.388681561Z namespaces/openshift-ingress/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.388724313Z namespaces/openshift-ingress/k8s.ovn.org/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.388758415Z namespaces/openshift-ingress/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.388842969Z namespaces/openshift-ingress/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.388888621Z namespaces/openshift-ingress/monitoring.coreos.com/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.388930333Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.388969915Z namespaces/openshift-ingress/networking.k8s.io/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.389042038Z namespaces/openshift-ingress/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.389109511Z namespaces/openshift-ingress/pods/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.389119982Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-46d7p/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.389134362Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-46d7p/istiod-openshift-gateway-5677ccb49b-46d7p.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.389256938Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-46d7p/discovery/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.389265358Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-46d7p/discovery/discovery/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.389269439Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-46d7p/discovery/discovery/logs/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.38930112Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-46d7p/discovery/discovery/logs/current.log [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.409126204Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-46d7p/discovery/discovery/logs/previous.insecure.log [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.409179877Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-46d7p/discovery/discovery/logs/previous.log [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.409226839Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-xf9js/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.409275161Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-xf9js/openshift-ai-inference-openshift-default-8699b59bbc-xf9js.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.409380826Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-xf9js/istio-proxy/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.409388886Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-xf9js/istio-proxy/istio-proxy/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.409393077Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-xf9js/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.409420648Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-xf9js/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.41073988Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-xf9js/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.410821174Z namespaces/openshift-ingress/pods/router-default-d6d96c7dc-qc8qb/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.410873896Z namespaces/openshift-ingress/pods/router-default-d6d96c7dc-qc8qb/router-default-d6d96c7dc-qc8qb.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.410978461Z namespaces/openshift-ingress/pods/router-default-d6d96c7dc-qc8qb/router/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.410987372Z namespaces/openshift-ingress/pods/router-default-d6d96c7dc-qc8qb/router/router/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.410993502Z namespaces/openshift-ingress/pods/router-default-d6d96c7dc-qc8qb/router/router/logs/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.411038474Z namespaces/openshift-ingress/pods/router-default-d6d96c7dc-qc8qb/router/router/logs/current.log [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.41115712Z namespaces/openshift-ingress/pods/router-default-d6d96c7dc-qc8qb/router/router/logs/previous.insecure.log [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.411226713Z namespaces/openshift-ingress/pods/router-default-d6d96c7dc-qc8qb/router/router/logs/previous.log [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.411241304Z namespaces/openshift-ingress/policy/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.411307947Z namespaces/openshift-ingress/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.411352359Z namespaces/openshift-ingress/route.openshift.io/ [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.411401701Z namespaces/openshift-ingress/route.openshift.io/routes.yaml [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.416454389Z [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.41646907Z sent 2,840 bytes received 1,127,195 bytes 2,260,070.00 bytes/sec [must-gather] [must-gather-5w2ht] OUT 2026-07-29T12:12:58.416482701Z total size is 17,408,214 speedup is 15.41 [must-gather] [must-gather ] OUT 2026-07-29T12:12:58.464809077Z namespace/openshift-must-gather-hkldr deleted [must-gather] [must-gather] [must-gather] Reprinting Cluster State: [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: 7e1b384b-0e13-46c5-87f6-afaf6fa56fd7 [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.26" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [must-gather] [must-gather ] OUT 2026-07-29T12:12:58.601401992Z Using must-gather plug-in image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:35f66492587d98a3cca9bf7ba4541c867770dc76b78b4cf9352e6715dca04502 [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: 7e1b384b-0e13-46c5-87f6-afaf6fa56fd7 [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.26" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [must-gather] [must-gather ] OUT 2026-07-29T12:12:58.6225704Z namespace/openshift-must-gather-jjmjm created [must-gather] [must-gather ] OUT 2026-07-29T12:12:58.628453277Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-vbslm created [must-gather] [must-gather ] OUT 2026-07-29T12:12:58.657119837Z pod for plug-in image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:35f66492587d98a3cca9bf7ba4541c867770dc76b78b4cf9352e6715dca04502 created [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:00.111562132Z [disk usage checker] Started [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:00.115456455Z [disk usage checker] Volume usage percentage: current = 20 ; allowed = 70 [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:00.349506286Z Gathering data for ns/openshift-cluster-version... [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:00.509894383Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:00.605430016Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:00.690762735Z Gathering data for ns/default... [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:01.035581789Z Gathering data for ns/openshift... [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:01.505301387Z Gathering data for ns/kube-system... [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:02.318362096Z Gathering data for ns/openshift-etcd... [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:02.659160961Z Wrote inspect data to must-gather. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:02.659195160Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:02.659195160Z namespaces "assisted-installer" not found [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:03.117952665Z Waiting on subprocesses to finish execution. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:03.125389528Z INFO: Gathering on-disk MachineConfig from degraded nodes [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:03.129699370Z Executing Istio gather script [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:03.139276815Z INFO: Gathering machine config daemon's old logs from all nodes [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:03.144382004Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:03.145021952Z INFO: Collecting host service logs for crio [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:03.145336803Z INFO: Collecting host service logs for kubelet [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:03.145609902Z INFO: Collecting host service logs for rpm-ostreed [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:03.145899161Z INFO: Collecting host service logs for ostree-finalize-staged [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:03.146197270Z INFO: Collecting host service logs for machine-config-daemon-firstboot [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:03.146511414Z INFO: Collecting host service logs for machine-config-daemon-host [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:03.146824569Z INFO: Collecting host service logs for NetworkManager [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:03.147143723Z INFO: Collecting host service logs for openvswitch [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:03.147493046Z INFO: Collecting host service logs for ovs-configuration [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:03.148090864Z INFO: Collecting host service logs for ovsdb-server [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:03.148491655Z INFO: Collecting host service logs for ovs-vswitchd [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:03.148904017Z INFO: Waiting for worker host service log collection to complete ... [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:03.159114083Z INFO: Gathering HAProxy config files [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:03.167959006Z INFO: Waiting for node performance related collection to complete ... [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:03.657045861Z No resources found in openshift-etcd namespace. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:03.785906394Z INFO: "metallb-operator" not detected. Skipping. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:04.065984569Z error: the server doesn't have a resource type "performanceprofile" [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:04.126960055Z INFO: "kubernetes-nmstate-operator" not detected. Skipping. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:04.207266596Z INFO: "sriov-network-operator" not detected. Skipping. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:04.864846658Z INFO: Waiting for on-disk MachineConfig collection to complete ... [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:04.864883866Z INFO: on-disk MachineConfig config collection complete. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:04.866049175Z No resources found [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:05.078665943Z No resources found [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:05.234116272Z INFO: namespace openshift-frr-k8s not detected. Skipping. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:05.363979431Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:05.658194852Z error: the server doesn't have a resource type "multi-networkpolicy" [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:05.791394382Z [disk usage checker] Volume usage percentage: current = 20 ; allowed = 70 [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:05.931257197Z Wrote inspect data to must-gather. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:05.970295066Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:05.984474364Z Wrote inspect data to must-gather. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:06.103234704Z INFO: Collecting Insights Archives from insights-operator-5885fdb759-5dgxd [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:06.103286877Z insights-runtime-extractor-2df6f [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:06.103307737Z insights-runtime-extractor-hbvl8 [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:06.103327978Z insights-runtime-extractor-q6pkk [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:06.137979683Z ERROR: No running kube-apiserver pods found [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:06.156341600Z Wrote inspect data to must-gather. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:06.159076371Z INFO: Found 1 replicas - prometheus-k8s-0 [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:06.159427804Z Wrote inspect data to must-gather. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:06.169345071Z INFO: Worker host service log collection to complete. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:06.229294872Z Wrote inspect data to must-gather. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:06.374009818Z INFO: Waiting for HAProxy config collection to complete ... [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:06.423452588Z Wrote inspect data to must-gather. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:06.435069358Z Inspecting resource clusterrole.rbac.authorization.k8s.io/istio-reader-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:06.580610171Z INFO: 'previous-logs' folder not found on ip-10-0-129-80.ec2.internal, skipping... [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:06.629125275Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:06.629125275Z See 'oc rsync -h' for help and examples [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:06.630414379Z INFO: Getting alertmanagers from prometheus-k8s-0 [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:06.724599890Z tar: Removing leading `/' from member names [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:06.734046608Z INFO: HAProxy config collection complete. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:06.774556238Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:06.790196176Z error: the server doesn't have a resource type "machineconfigs" [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:06.968081836Z Wrote inspect data to must-gather. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:07.026474563Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:07.037070469Z Inspecting resource clusterrole.rbac.authorization.k8s.io/istiod-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:07.099578434Z INFO: 'previous-logs' folder not found on ip-10-0-133-86.ec2.internal, skipping... [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:07.194594614Z error: the server doesn't have a resource type "machineconfigpools" [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:07.289934418Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:07.434810335Z INFO: Getting rules from prometheus-k8s-0 [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:07.546309918Z Wrote inspect data to must-gather. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:07.624644463Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:07.634126296Z Inspecting resource clusterrole.rbac.authorization.k8s.io/istiod-gateway-controller-openshift-gateway-openshift-ingress [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:07.679697066Z INFO: 'previous-logs' folder not found on ip-10-0-140-240.ec2.internal, skipping... [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:07.683632382Z INFO: Waiting for Machine Config Daemon termination log collection to complete ... [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:07.683708647Z INFO: Machine Config Daemon termination log collection complete. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:07.721694403Z Gathering data for ns/openshift-monitoring... [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:07.799429751Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:07.819405453Z Wrote inspect data to must-gather. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:08.037338074Z Wrote inspect data to must-gather. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:08.038520330Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:08.042084612Z Inspecting resource clusterrolebinding.rbac.authorization.k8s.io/istio-reader-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:08.048536376Z INFO: Getting status/config from prometheus-k8s-0 [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:08.073917265Z error: the server doesn't have a resource type "kubeletconfigs" [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:08.349668670Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:08.354194269Z Inspecting resource clusterrolebinding.rbac.authorization.k8s.io/istiod-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:08.431437324Z Wrote inspect data to must-gather. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:08.459644852Z INFO: INTERCONNECT MODE [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:08.459730117Z INFO: Gathering ovn-kubernetes DBs [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:08.574843164Z INFO: Getting status/flags from prometheus-k8s-0 [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:08.658960414Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:08.661862183Z Inspecting resource clusterrolebinding.rbac.authorization.k8s.io/istiod-gateway-controller-openshift-gateway-openshift-ingress [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:08.712117734Z INFO: Gathering OVN_Northbound from ovnkube-node-bwrjz... [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:08.712406169Z INFO: Gathering OVN_Northbound from ovnkube-node-jkg9m... [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:08.712653028Z INFO: Gathering OVN_Northbound from ovnkube-node-tpvc8... [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:08.712904097Z INFO: Gathering OVN_Southbound from ovnkube-node-bwrjz... [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:08.713129670Z INFO: Gathering OVN_Southbound from ovnkube-node-jkg9m... [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:08.713399667Z INFO: Gathering OVN_Southbound from ovnkube-node-tpvc8... [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:08.818435354Z Wrote inspect data to must-gather. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:08.924987165Z INFO: Getting status/runtimeinfo from prometheus-k8s-0 [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:09.017431525Z tar: Removing leading `/' from member names [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:09.072408940Z tar: Removing leading `/' from member names [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:09.072857105Z tar: Removing leading `/' from member names [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:09.162395610Z tar: Removing leading `/' from member names [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:09.175448712Z tar: Removing leading `/' from member names [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:09.180416487Z tar: Removing leading `/' from member names [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:09.201087831Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:09.378611971Z INFO: Getting targets?state=active from prometheus-k8s-0 [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:09.387987820Z Wrote inspect data to must-gather. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:09.714419682Z INFO: Getting status/tsdb from prometheus-k8s-0 [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:09.751995041Z Wrote inspect data to must-gather. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:09.878255224Z INFO: Waiting for network log collection to complete ... [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:09.879554269Z INFO: Waiting for ovnk database copies to complete ... [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:09.879554269Z INFO: Copying ovnk databases complete. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:09.907407500Z 6.9M must-gather/network_logs/ovnk_database_store [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:09.910492041Z ovnk_database_store/ [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:09.910550741Z ovnk_database_store/ovnkube-node-bwrjz_nbdb [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:09.929030800Z ovnk_database_store/ovnkube-node-tpvc8_sbdb [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:09.977804180Z ovnk_database_store/ovnkube-node-jkg9m_nbdb [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:09.997755785Z ovnk_database_store/ovnkube-node-tpvc8_nbdb [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:10.015025888Z ovnk_database_store/ovnkube-node-jkg9m_sbdb [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:10.064308845Z ovnk_database_store/ovnkube-node-bwrjz_sbdb [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:10.101896688Z W0729 12:13:10.101841 1566 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in '' [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:10.109130345Z Wrote inspect data to must-gather. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:10.113232975Z INFO: Network log collection complete. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:10.205346149Z INFO: Getting status from alertmanager-main-0 [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:10.488623213Z Gathering data for ns/openshift-network-console... [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:10.554604556Z Wrote inspect data to must-gather. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:10.802547084Z [disk usage checker] Volume usage percentage: current = 21 ; allowed = 70 [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:10.976895098Z Wrote inspect data to must-gather. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:11.117541806Z Gathering data for ns/openshift-console-operator... [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:11.189266973Z Wrote inspect data to must-gather. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:11.305143045Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:11.307812842Z INFO: Fallback to identify the container image from release info [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:11.563711248Z Gathering data for ns/openshift-console... [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:11.638076870Z Inspecting resource crd/authorizationpolicies.security.istio.io [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:11.847772297Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:12.097464950Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:12.101035609Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811 [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:12.101101267Z Inspecting resource crd/destinationrules.networking.istio.io [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:12.158681924Z Gathering data for ns/openshift-cluster-storage-operator... [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:12.247912073Z daemonset.apps/perf-node-gather-daemonset created [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:12.394152266Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:12.411004529Z Waiting for performance profile collector pods to become ready: 1 [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:12.643499602Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:12.647207884Z Inspecting resource crd/envoyfilters.networking.istio.io [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:12.831805044Z Gathering data for ns/openshift-dns-operator... [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:12.911819220Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:13.148674300Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:13.151775892Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:13.186322907Z Gathering data for ns/openshift-dns... [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:13.327518493Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:13.555152969Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:13.557606950Z Inspecting resource crd/gateways.gateway.networking.k8s.io [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:13.572548752Z Waiting for performance profile collector pods to become ready: 2 [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:13.783037196Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:13.880536069Z Gathering data for ns/openshift-image-registry... [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:13.996497473Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:13.999005932Z Inspecting resource crd/gateways.networking.istio.io [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:14.191578892Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:14.417771353Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:14.420478393Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:14.461134451Z Gathering data for ns/openshift-ingress-operator... [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:14.649872120Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:14.737806817Z Waiting for performance profile collector pods to become ready: 3 [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:14.879912752Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:14.886587810Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:14.888631215Z Inspecting resource crd/httproutes.gateway.networking.k8s.io [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:15.100552090Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:15.306248213Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:15.308694081Z Inspecting resource crd/inferencepools.inference.networking.k8s.io [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:15.480543474Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:15.714757691Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:15.717439640Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:15.813581633Z [disk usage checker] Volume usage percentage: current = 21 ; allowed = 70 [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:15.883839176Z Waiting for performance profile collector pods to become ready: 4 [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:15.942968140Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:15.950716003Z Gathering data for ns/openshift-ingress-canary... [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:16.161835328Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:16.164406312Z Inspecting resource crd/peerauthentications.security.istio.io [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:16.363924400Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:16.414676222Z Gathering data for ns/openshift-insights... [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:16.563718907Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:16.566433567Z Inspecting resource crd/proxyconfigs.networking.istio.io [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:16.754869998Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:16.981354979Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:16.984871282Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:17.052679625Z Waiting for performance profile collector pods to become ready: 5 [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:17.186567171Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:17.388929088Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:17.391143313Z Inspecting resource crd/requestauthentications.security.istio.io [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:17.572239482Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:17.767335996Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:17.769976394Z Inspecting resource crd/serviceentries.networking.istio.io [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:17.955180844Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:18.186129622Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:18.190519621Z Inspecting resource crd/sidecars.networking.istio.io [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:18.198678041Z Waiting for performance profile collector pods to become ready: 6 [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:18.369278043Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:18.579571631Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:18.582933849Z Inspecting resource crd/telemetries.telemetry.istio.io [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:18.771658451Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:18.981203328Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:18.983854552Z Inspecting resource crd/virtualservices.networking.istio.io [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:19.189287000Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:19.347441898Z Waiting for performance profile collector pods to become ready: 7 [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:19.414914287Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:19.417953711Z Inspecting resource crd/wasmplugins.extensions.istio.io [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:19.499909968Z Gathering data for ns/openshift-lws-operator... [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:19.601557984Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:19.833340386Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:19.837160721Z Inspecting resource crd/workloadentries.networking.istio.io [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:20.044572398Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:20.125952057Z Gathering data for ns/kserve... [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:20.247529194Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:20.249758406Z Inspecting resource crd/workloadgroups.networking.istio.io [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:20.449581913Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:20.494888384Z Daemonset perf-node-gather-daemonset ready 3 out of 3 [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:20.676118284Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:20.790882510Z Collecting performance related data for node ip-10-0-129-80.ec2.internal [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:20.801870941Z Collecting performance related data for node ip-10-0-140-240.ec2.internal [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:20.813481706Z Collecting performance related data for node ip-10-0-133-86.ec2.internal [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:20.815090308Z Inspecting resource mutatingwebhookconfiguration.admissionregistration.k8s.io/istio-sidecar-injector-openshift-gateway-openshift-ingress [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:20.820595575Z [disk usage checker] Volume usage percentage: current = 21 ; allowed = 70 [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:21.059101218Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:21.970428428Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:22.208092228Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:22.410994273Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:22.595385511Z Inspecting resource validatingwebhookconfiguration.admissionregistration.k8s.io/istio-validator-openshift-gateway-openshift-ingress [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:22.834001898Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:23.519348283Z Gathering data for ns/openshift-config... [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:23.629807785Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:23.852718752Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:23.974606653Z Gathering data for ns/openshift-config-managed... [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:24.031529086Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:24.601951216Z Done executing Istio gather script [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:24.605606937Z Gathering data for ns/openshift-kube-apiserver-operator... [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:24.919899974Z Gathering data for ns/openshift-kube-apiserver... [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:25.215809068Z Gathering data for ns/openshift-kube-controller-manager... [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:25.588158435Z Gathering data for ns/openshift-kube-controller-manager-operator... [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:25.826889811Z [disk usage checker] Volume usage percentage: current = 21 ; allowed = 70 [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:25.944615487Z Gathering data for ns/openshift-kube-scheduler... [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:26.257532128Z Gathering data for ns/openshift-kube-scheduler-operator... [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:26.629985024Z Gathering data for ns/openshift-kube-storage-version-migrator... [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:26.998072623Z Gathering data for ns/openshift-kube-storage-version-migrator-operator... [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:27.356582164Z Gathering data for ns/openshift-user-workload-monitoring... [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:27.999479646Z Gathering data for ns/openshift-multus... [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:29.480575978Z Collecting kubelet logs for node ip-10-0-129-80.ec2.internal [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:29.482506283Z Collecting kubelet logs for node ip-10-0-140-240.ec2.internal [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:29.485699518Z Collecting kubelet logs for node ip-10-0-133-86.ec2.internal [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:29.591385020Z Gathering data for ns/openshift-ovn-kubernetes... [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:29.909267613Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-jjmjm namespace [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:29.914174635Z INFO: Node performance data collection complete. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:30.834917693Z [disk usage checker] Volume usage percentage: current = 21 ; allowed = 70 [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:30.990698954Z Gathering data for ns/openshift-host-network... [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:31.337330158Z Gathering data for ns/openshift-network-diagnostics... [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:31.904629988Z Gathering data for ns/openshift-network-node-identity... [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:32.335787375Z Gathering data for ns/openshift-network-operator... [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:32.721908022Z Gathering data for ns/openshift-cloud-network-config-controller... [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:33.006883028Z Gathering data for ns/openshift-cluster-node-tuning-operator... [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:33.512091793Z Gathering data for ns/openshift-apiserver-operator... [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:33.821583114Z Gathering data for ns/openshift-apiserver... [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:34.281777320Z Gathering data for ns/openshift-controller-manager-operator... [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:34.585551714Z Gathering data for ns/openshift-controller-manager... [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:34.916363566Z Gathering data for ns/openshift-cluster-samples-operator... [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:35.593236460Z Gathering data for ns/openshift-operator-lifecycle-manager... [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:35.841938157Z [disk usage checker] Volume usage percentage: current = 21 ; allowed = 70 [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:35.897513317Z Gathering data for ns/openshift-service-ca-operator... [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:36.232523637Z Gathering data for ns/openshift-service-ca... [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:36.607847904Z Gathering data for ns/openshift-cluster-csi-drivers... [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:37.553527583Z Wrote inspect data to must-gather. [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:37.553574784Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:37.553574784Z skipping gathering secrets/support due to error: secrets "support" not found [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:37.692072818Z error: the server doesn't have a resource type "clusters" [must-gather] [must-gather-cwdzq] POD 2026-07-29T12:13:37.872531400Z Caches written to disk [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:38.884465613Z waiting for gather to complete [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:38.887673984Z downloading gather output [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.122686831Z receiving incremental file list [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.141863194Z ./ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.141957058Z aggregated-discovery-api.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.142290304Z aggregated-discovery-apis.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.143183616Z event-filter.html [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.146143565Z timestamp [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.14626427Z version [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.153716231Z cluster-scoped-resources/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.153736452Z cluster-scoped-resources/admissionregistration.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.153742542Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.153803595Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/default-network-annotation.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.153940032Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/servicecidrs.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.15412182Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/user-defined-networks-namespace-label.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.154230605Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.154297739Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/default-network-annotation-binding.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.154425975Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/servicecidrs-binding.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.15454477Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/user-defined-networks-namespace-label-binding.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.154661416Z cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.154716858Z cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/multus.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.154842104Z cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/network-node-identity.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.154934818Z cluster-scoped-resources/apiextensions.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.155432482Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.155489555Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/accounts.nim.opendatahub.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.155719395Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/adminnetworkpolicies.policy.networking.k8s.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.155973447Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/adminpolicybasedexternalroutes.k8s.ovn.org.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.156258151Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertingrules.monitoring.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.1564444Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertmanagerconfigs.monitoring.coreos.com.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.160259089Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertmanagers.monitoring.coreos.com.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.161885565Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertrelabelconfigs.monitoring.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.162074945Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apikeyapprovals.devportal.kuadrant.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.162225652Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apikeyrequests.devportal.kuadrant.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.162363438Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apikeys.devportal.kuadrant.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.162512725Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apiproducts.devportal.kuadrant.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.16304107Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apirequestcounts.apiserver.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.163196507Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apiservers.config.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.163373606Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/appliedmanifestworks.work.open-cluster-management.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.163487641Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authconfigs.authorino.kuadrant.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.164400924Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authentications.config.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.164649876Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authorinos.operator.authorino.kuadrant.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.164767981Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authorizationpolicies.security.istio.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.164926028Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authpolicies.kuadrant.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.166212719Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/baselineadminnetworkpolicies.policy.networking.k8s.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.166466921Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/builds.config.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.16665262Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/bundles.trust.cert-manager.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.166838479Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/catalogsources.operators.coreos.com.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.167182925Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/certificaterequests.cert-manager.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.167366943Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/certificates.cert-manager.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.167581244Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/certmanagers.operator.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.167881488Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/challenges.acme.cert-manager.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.168651574Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/cloudcredentials.operator.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.168792781Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/cloudeventsources.eventing.keda.sh.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.168921527Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/cloudprivateipconfigs.cloud.network.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.169075144Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterclaims.cluster.open-cluster-management.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.169193509Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clustercloudeventsources.eventing.keda.sh.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.169317065Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clustercsidrivers.operator.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.169520285Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterimagepolicies.config.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.169717544Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterissuers.cert-manager.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.170609546Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusteroperators.config.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.170754933Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterresourcequotas.quota.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.170884939Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterserviceversions.operators.coreos.com.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.172909234Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterstoragecontainers.serving.kserve.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.173102803Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clustertriggerauthentications.keda.sh.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.173257371Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusteruserdefinednetworks.k8s.ovn.org.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.173487962Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterversions.config.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.173771805Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/configs.imageregistry.operator.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.174240747Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/configs.operator.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.174365673Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/configs.samples.operator.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.174487909Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleclidownloads.console.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.174628275Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleexternalloglinks.console.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.17474521Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolelinks.console.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.174904628Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolenotifications.console.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.175048935Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleplugins.console.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.175246254Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolequickstarts.console.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.175443634Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoles.config.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.175569989Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoles.operator.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.175844392Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolesamples.console.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.175993499Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleyamlsamples.console.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.176127296Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/controllerconfigs.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.176680412Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/credentialsrequests.cloudcredential.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.176803158Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/csisnapshotcontrollers.operator.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.176937694Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/datascienceclusters.datasciencecluster.opendatahub.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.177148594Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/destinationrules.networking.istio.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.177899799Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnses.config.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.178022155Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnses.operator.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.178239205Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnshealthcheckprobes.kuadrant.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.17835225Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnspolicies.kuadrant.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.178511258Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnsrecords.ingress.operator.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.178627783Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnsrecords.kuadrant.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.17877971Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dscinitializations.dscinitialization.opendatahub.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.178912167Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressfirewalls.k8s.ovn.org.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.179057624Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressips.k8s.ovn.org.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.179178689Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressqoses.k8s.ovn.org.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.179312665Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressrouters.network.operator.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.179438771Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressservices.k8s.ovn.org.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.179557957Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/envoyfilters.networking.istio.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.179697534Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/etcds.operator.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.17983136Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/featuregates.config.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.180095032Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gatewayclasses.gateway.networking.k8s.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.18025547Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.gateway.networking.k8s.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.180612827Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.networking.istio.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.180783875Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/grpcroutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.182470224Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/helmchartrepositories.helm.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.182575839Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/httproutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.183413858Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/httpscaledobjects.http.keda.sh.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.183549525Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagecontentpolicies.config.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.183678101Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagecontentsourcepolicies.operator.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.183787296Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagedigestmirrorsets.config.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.183914502Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagepolicies.config.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.184195255Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagepruners.imageregistry.operator.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.184484339Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/images.config.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.184619735Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagetagmirrorsets.config.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.1847281Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencegraphs.serving.kserve.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.184881178Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencemodelrewrites.llm-d.ai.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.185000693Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferenceobjectives.llm-d.ai.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.18514055Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.k8s.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.185284477Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.185408492Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferenceservices.serving.kserve.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.189493664Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/infrastructures.config.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.189931495Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ingresscontrollers.operator.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.190512492Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ingresses.config.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.190692621Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/insightsoperators.operator.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.190857459Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/installplans.operators.coreos.com.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.190997475Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/interceptorroutes.http.keda.sh.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.191155393Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ipamclaims.k8s.cni.cncf.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.191259918Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ippools.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.191376753Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/issuers.cert-manager.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.192195122Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/istiocsrs.operator.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.192510946Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kedacontrollers.keda.sh.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.196957026Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kuadrants.kuadrant.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.197106683Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubeapiservers.operator.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.197244309Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubecontrollermanagers.operator.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.197380086Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubeschedulers.operator.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.197504312Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubestorageversionmigrators.operator.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.197616607Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/leaderworkersetoperators.operator.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.197728662Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/leaderworkersets.leaderworkerset.x-k8s.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.200411518Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/limitadors.limitador.kuadrant.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.205953069Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/llminferenceserviceconfigs.serving.kserve.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.21106874Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/llminferenceservices.serving.kserve.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.216468404Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineconfignodes.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.216623281Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineconfigurations.operator.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.216903754Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineosbuilds.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.217086743Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineosconfigs.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.21724345Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/network-attachment-definitions.k8s.cni.cncf.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.217345345Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/networks.config.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.217553665Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/networks.operator.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.21787552Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/nodes.config.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.218049498Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/nodeslicepools.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.218202005Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/oauths.config.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.218413505Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/oidcpolicies.extensions.kuadrant.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.218542861Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/olmconfigs.operators.coreos.com.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.218648516Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/openshiftapiservers.operator.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.218766892Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/openshiftcontrollermanagers.operator.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.218878527Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorconditions.operators.coreos.com.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.219029664Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorgroups.operators.coreos.com.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.219194202Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorhubs.config.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.219317258Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorpkis.network.operator.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.219421443Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operators.operators.coreos.com.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.219548349Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/orders.acme.cert-manager.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.219707456Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/overlappingrangeipreservations.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.219807511Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/peerauthentications.security.istio.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.219936927Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/pinnedimagesets.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.220071293Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/planpolicies.extensions.kuadrant.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.22020797Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/podmonitors.monitoring.coreos.com.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.220477682Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/podnetworkconnectivitychecks.controlplane.operator.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.220607468Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/probes.monitoring.coreos.com.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.22085935Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/profiles.tuned.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.220968015Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/projecthelmchartrepositories.helm.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.221099751Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/projects.config.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.221206507Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/prometheuses.monitoring.coreos.com.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.223254853Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/prometheusrules.monitoring.coreos.com.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.223385879Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/proxies.config.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.223495104Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/proxyconfigs.networking.istio.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.223600919Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/rangeallocations.security.internal.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.223708844Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ratelimitpolicies.kuadrant.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.223857361Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/referencegrants.gateway.networking.k8s.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.223976507Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/requestauthentications.security.istio.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.224149745Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/rolebindingrestrictions.authorization.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.224275031Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/scaledjobs.keda.sh.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.226021583Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/scaledobjects.keda.sh.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.226178421Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/schedulers.config.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.226309797Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/securitycontextconstraints.security.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.226451703Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/servicecas.operator.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.226568269Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/serviceentries.networking.istio.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.226754728Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/servicemonitors.monitoring.coreos.com.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.226995509Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/servingruntimes.serving.kserve.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.227531794Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/sidecars.networking.istio.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.227796607Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/storages.operator.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.227912022Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/storagestates.migration.k8s.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.228039228Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/storageversionmigrations.migration.k8s.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.228160934Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/subscriptions.operators.coreos.com.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.228782473Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/telemetries.telemetry.istio.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.228962092Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/telemetrypolicies.extensions.kuadrant.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.229098368Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/thanosrulers.monitoring.coreos.com.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.230584898Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/tlspolicies.kuadrant.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.230764316Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/tokenratelimitpolicies.kuadrant.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.230952535Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/trainedmodels.serving.kserve.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.231078971Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/triggerauthentications.keda.sh.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.231245699Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/trustmanagers.operator.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.231528642Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/tuneds.tuned.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.231648348Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/userdefinednetworks.k8s.ovn.org.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.231822006Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/virtualservices.networking.istio.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.232241026Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumepopulators.populator.storage.k8s.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.232348621Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumesnapshotclasses.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.232455326Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumesnapshotcontents.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.232629294Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumesnapshots.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.232776721Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/wasmplugins.extensions.istio.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.232907117Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadentries.networking.istio.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.233064345Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadgroups.networking.istio.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.233214092Z cluster-scoped-resources/apiregistration.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.233487745Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.233548327Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1..yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.233633681Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.acme.cert-manager.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.233714045Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.admissionregistration.k8s.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.233797599Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apiextensions.k8s.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.233880003Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apiserver.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.234061032Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apps.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.234163106Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apps.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.23424916Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.authentication.k8s.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.234337325Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.authorization.k8s.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.234422578Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.authorization.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.234513433Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.autoscaling.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.234590996Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.batch.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.234677381Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.build.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.234767975Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.cert-manager.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.234862619Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.certificates.k8s.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.234941743Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.cloud.network.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.235053018Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.cloudcredential.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.235141622Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.config.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.235227536Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.console.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.235331821Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.coordination.k8s.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.235420775Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.datasciencecluster.opendatahub.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.2355074Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.discovery.k8s.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.235611694Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.dscinitialization.opendatahub.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.23573525Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.events.k8s.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.235860716Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.flowcontrol.apiserver.k8s.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.235985042Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.gateway.networking.k8s.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.236143619Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.image.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.236235704Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.imageregistry.operator.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.236323948Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.inference.networking.k8s.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.236409092Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.ingress.operator.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.236496876Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.k8s.cni.cncf.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.23658607Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.k8s.ovn.org.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.236668784Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.kuadrant.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.236756678Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.leaderworkerset.x-k8s.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.236844102Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.236932796Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.monitoring.coreos.com.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.237021411Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.monitoring.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.237128366Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.network.operator.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.23721479Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.networking.istio.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.237304814Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.networking.k8s.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.237386818Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.nim.opendatahub.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.237474302Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.node.k8s.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.237559956Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.oauth.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.237654261Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.operator.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.237740134Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.operators.coreos.com.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.237828469Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.packages.operators.coreos.com.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.237924253Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.policy.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.23806619Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.project.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.238157124Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.quota.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.238251578Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.rbac.authorization.k8s.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.238346653Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.resource.k8s.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.238429147Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.route.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.238526071Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.samples.operator.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.238614726Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.scheduling.k8s.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.238809995Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.security.internal.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.238897339Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.security.istio.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.238989743Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.security.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.239118189Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.239199623Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.storage.k8s.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.239297198Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.telemetry.istio.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.239410893Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.template.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.239527358Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.tuned.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.239638474Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.user.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.23976744Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.work.open-cluster-management.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.239904876Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.cluster.open-cluster-management.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.240061904Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.controlplane.operator.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.24019751Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.devportal.kuadrant.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.240308145Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.eventing.keda.sh.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.240439071Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.extensions.istio.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.240533036Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.extensions.kuadrant.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.24062335Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.http.keda.sh.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.240715845Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.k8s.cni.cncf.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.240810739Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.keda.sh.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.240896763Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.kuadrant.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.240993098Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.limitador.kuadrant.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.241106573Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.migration.k8s.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.241191807Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.monitoring.coreos.com.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.241284931Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.operator.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.241374965Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.operators.coreos.com.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.24147412Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.policy.networking.k8s.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.241558914Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.serving.kserve.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.241655049Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.telemetry.istio.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.241740663Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.trust.cert-manager.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.241836547Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.241928951Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.242021826Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.llm-d.ai.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.242129941Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.operators.coreos.com.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.242218025Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.serving.kserve.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.242310789Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha3.networking.istio.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.242398294Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.admissionregistration.k8s.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.242487018Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.external.metrics.k8s.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.242599513Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.gateway.networking.k8s.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.242682737Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.helm.openshift.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.242774231Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.http.keda.sh.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.242869316Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.kuadrant.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.24295345Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.metrics.k8s.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.243075605Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.monitoring.coreos.com.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.24316605Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.networking.istio.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.243262104Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.operator.authorino.kuadrant.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.243362209Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.populator.storage.k8s.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.243456694Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.security.istio.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.243544758Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.serving.kserve.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.243639752Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.storage.k8s.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.243724756Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta3.authorino.kuadrant.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.24380848Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v2.autoscaling.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.243898954Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v2.operators.coreos.com.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.243948977Z cluster-scoped-resources/certificates.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.243962027Z cluster-scoped-resources/certificates.k8s.io/certificatesigningrequests/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.24401888Z cluster-scoped-resources/certificates.k8s.io/certificatesigningrequests/system:openshift:openshift-monitoring-6xwtd.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.244166717Z cluster-scoped-resources/certificates.k8s.io/certificatesigningrequests/system:openshift:openshift-monitoring-kmdhh.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.244292893Z cluster-scoped-resources/certificates.k8s.io/certificatesigningrequests/system:openshift:openshift-monitoring-tg87l.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.244546505Z cluster-scoped-resources/config.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.244600457Z cluster-scoped-resources/config.openshift.io/apiservers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.244796116Z cluster-scoped-resources/config.openshift.io/authentications.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.244967054Z cluster-scoped-resources/config.openshift.io/builds.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.24508893Z cluster-scoped-resources/config.openshift.io/clusterimagepolicies.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.245163824Z cluster-scoped-resources/config.openshift.io/clusteroperators.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.245474448Z cluster-scoped-resources/config.openshift.io/clusterversions.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.245579163Z cluster-scoped-resources/config.openshift.io/consoles.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.245670607Z cluster-scoped-resources/config.openshift.io/dnses.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.245754482Z cluster-scoped-resources/config.openshift.io/featuregates.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.245858376Z cluster-scoped-resources/config.openshift.io/imagecontentpolicies.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.24593956Z cluster-scoped-resources/config.openshift.io/imagedigestmirrorsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.246050276Z cluster-scoped-resources/config.openshift.io/imagepolicies.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.2461382Z cluster-scoped-resources/config.openshift.io/images.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.246224364Z cluster-scoped-resources/config.openshift.io/imagetagmirrorsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.246308137Z cluster-scoped-resources/config.openshift.io/infrastructures.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.246419913Z cluster-scoped-resources/config.openshift.io/ingresses.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.246532498Z cluster-scoped-resources/config.openshift.io/networks.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.246645213Z cluster-scoped-resources/config.openshift.io/nodes.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.246747968Z cluster-scoped-resources/config.openshift.io/oauths.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.246850363Z cluster-scoped-resources/config.openshift.io/operatorhubs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.246954858Z cluster-scoped-resources/config.openshift.io/projects.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.247069984Z cluster-scoped-resources/config.openshift.io/proxies.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.247175398Z cluster-scoped-resources/config.openshift.io/schedulers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.247285114Z cluster-scoped-resources/config.openshift.io/clusteroperators/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.247340176Z cluster-scoped-resources/config.openshift.io/clusteroperators/console.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.247475743Z cluster-scoped-resources/config.openshift.io/clusteroperators/csi-snapshot-controller.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.247596818Z cluster-scoped-resources/config.openshift.io/clusteroperators/dns.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.247722634Z cluster-scoped-resources/config.openshift.io/clusteroperators/image-registry.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.24784785Z cluster-scoped-resources/config.openshift.io/clusteroperators/ingress.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.247969026Z cluster-scoped-resources/config.openshift.io/clusteroperators/insights.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.248096662Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-apiserver.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.248256989Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-controller-manager.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.248366525Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-scheduler.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.248470119Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-storage-version-migrator.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.248579624Z cluster-scoped-resources/config.openshift.io/clusteroperators/monitoring.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.24868959Z cluster-scoped-resources/config.openshift.io/clusteroperators/network.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.248852827Z cluster-scoped-resources/config.openshift.io/clusteroperators/node-tuning.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.248967443Z cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-apiserver.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.249089608Z cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-controller-manager.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.249199344Z cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-samples.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.249309199Z cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager-catalog.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.249412014Z cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager-packageserver.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.249521419Z cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.249623024Z cluster-scoped-resources/config.openshift.io/clusteroperators/service-ca.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.249734599Z cluster-scoped-resources/config.openshift.io/clusteroperators/storage.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.249813022Z cluster-scoped-resources/config.openshift.io/clusterversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.249882066Z cluster-scoped-resources/config.openshift.io/clusterversions/version.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.249954589Z cluster-scoped-resources/config.openshift.io/consoles/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.250029233Z cluster-scoped-resources/config.openshift.io/consoles/cluster.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.250094466Z cluster-scoped-resources/config.openshift.io/featuregates/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.250151138Z cluster-scoped-resources/config.openshift.io/featuregates/cluster.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.250221782Z cluster-scoped-resources/config.openshift.io/infrastructures/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.250281795Z cluster-scoped-resources/config.openshift.io/infrastructures/cluster.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.250349868Z cluster-scoped-resources/config.openshift.io/oauths/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.250417731Z cluster-scoped-resources/config.openshift.io/oauths/cluster.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.250484194Z cluster-scoped-resources/config.openshift.io/proxies/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.250545557Z cluster-scoped-resources/config.openshift.io/proxies/cluster.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.250626281Z cluster-scoped-resources/console.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.250636971Z cluster-scoped-resources/console.openshift.io/consoleplugins/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.250692834Z cluster-scoped-resources/console.openshift.io/consoleplugins/kuadrant-console-plugin.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.25081304Z cluster-scoped-resources/console.openshift.io/consoleplugins/monitoring-plugin.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.250928985Z cluster-scoped-resources/console.openshift.io/consoleplugins/networking-console-plugin.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.251021709Z cluster-scoped-resources/core/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.251064911Z cluster-scoped-resources/core/nodes/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.251108954Z cluster-scoped-resources/core/nodes/ip-10-0-129-80.ec2.internal.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.251419668Z cluster-scoped-resources/core/nodes/ip-10-0-133-86.ec2.internal.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.251570865Z cluster-scoped-resources/core/nodes/ip-10-0-140-240.ec2.internal.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.251698171Z cluster-scoped-resources/core/persistentvolumes/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.251748724Z cluster-scoped-resources/core/persistentvolumes/pvc-b8b0e167-4bf3-4d8d-86ab-b583ee0ba857.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.251859769Z cluster-scoped-resources/core/persistentvolumes/pvc-ce3748df-ddb8-4bce-a255-9229c58859db.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.251925372Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.252022777Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.252133532Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/catch-all.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.252268908Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/exempt.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.252352612Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/global-default.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.252438856Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/kube-controller-manager.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.25251923Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/kube-scheduler.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.252612984Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/kube-system-service-accounts.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.252851785Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-apiserver-operator.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.25294744Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-apiserver-sar.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.253101377Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-apiserver.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.253251794Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-authentication-operator.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.253339559Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-controller-manager.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.253430863Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-kube-apiserver-operator.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.253523917Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-monitoring-metrics.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.253613221Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-oauth-apiserver-sar.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.253702165Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-oauth-apiserver.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.253785659Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-oauth-server.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.253874693Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/probes.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.253969428Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/service-accounts.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.254084853Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/system-leader-election.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.254171238Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/system-node-high.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.254261992Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/system-nodes.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.254329765Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.254379467Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/catch-all.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.254456911Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/exempt.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.254536905Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/global-default.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.254620449Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/leader-election.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.254699192Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/node-high.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.254785546Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/openshift-control-plane-operators.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.254933113Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/system.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.255093631Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/workload-high.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.255185725Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/workload-low.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.255233337Z cluster-scoped-resources/gateway.networking.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.255246128Z cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.255331732Z cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/openshift-default.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.255466288Z cluster-scoped-resources/imageregistry.operator.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.255475769Z cluster-scoped-resources/imageregistry.operator.openshift.io/configs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.255563933Z cluster-scoped-resources/imageregistry.operator.openshift.io/configs/cluster.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.255701439Z cluster-scoped-resources/imageregistry.operator.openshift.io/imagepruners/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.255732721Z cluster-scoped-resources/imageregistry.operator.openshift.io/imagepruners/cluster.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.255806564Z cluster-scoped-resources/migration.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.255815565Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.255870398Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/console-plugin-storage-version-migration.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.255998833Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/machineconfiguration-controllerconfig-storage-version-migration.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.256221924Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/machineconfiguration-machineconfigpool-storage-version-migration.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.256289437Z cluster-scoped-resources/oauth.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.256320399Z cluster-scoped-resources/oauth.openshift.io/oauthclients/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.256374801Z cluster-scoped-resources/oauth.openshift.io/oauthclients/console.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.256490377Z cluster-scoped-resources/operator.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.256537399Z cluster-scoped-resources/operator.openshift.io/clustercsidrivers/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.256573371Z cluster-scoped-resources/operator.openshift.io/clustercsidrivers/ebs.csi.aws.com.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.256702177Z cluster-scoped-resources/operator.openshift.io/consoles/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.256745289Z cluster-scoped-resources/operator.openshift.io/consoles/cluster.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.256896696Z cluster-scoped-resources/operator.openshift.io/csisnapshotcontrollers/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.256937448Z cluster-scoped-resources/operator.openshift.io/csisnapshotcontrollers/cluster.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.257496044Z cluster-scoped-resources/operator.openshift.io/dnses/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.257520855Z cluster-scoped-resources/operator.openshift.io/dnses/default.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.257584598Z cluster-scoped-resources/operator.openshift.io/insightsoperators/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.257647801Z cluster-scoped-resources/operator.openshift.io/insightsoperators/cluster.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.257720445Z cluster-scoped-resources/operator.openshift.io/kubeapiservers/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.257814949Z cluster-scoped-resources/operator.openshift.io/kubeapiservers/cluster.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.257905863Z cluster-scoped-resources/operator.openshift.io/kubecontrollermanagers/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.257995758Z cluster-scoped-resources/operator.openshift.io/kubecontrollermanagers/cluster.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.258109223Z cluster-scoped-resources/operator.openshift.io/kubeschedulers/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.258171186Z cluster-scoped-resources/operator.openshift.io/kubeschedulers/cluster.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.258244819Z cluster-scoped-resources/operator.openshift.io/kubestorageversionmigrators/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.258332933Z cluster-scoped-resources/operator.openshift.io/kubestorageversionmigrators/cluster.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.258556854Z cluster-scoped-resources/operator.openshift.io/networks/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.258596766Z cluster-scoped-resources/operator.openshift.io/networks/cluster.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.258700781Z cluster-scoped-resources/operator.openshift.io/openshiftapiservers/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.258717281Z cluster-scoped-resources/operator.openshift.io/openshiftapiservers/cluster.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.258827957Z cluster-scoped-resources/operator.openshift.io/openshiftcontrollermanagers/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.258869319Z cluster-scoped-resources/operator.openshift.io/openshiftcontrollermanagers/cluster.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.258950922Z cluster-scoped-resources/operator.openshift.io/servicecas/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.259038316Z cluster-scoped-resources/operator.openshift.io/servicecas/cluster.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.259137581Z cluster-scoped-resources/operator.openshift.io/storages/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.259174893Z cluster-scoped-resources/operator.openshift.io/storages/cluster.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.259343511Z cluster-scoped-resources/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.259377343Z cluster-scoped-resources/operators.coreos.com/olmconfigs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.259409884Z cluster-scoped-resources/operators.coreos.com/olmconfigs/cluster.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.259497618Z cluster-scoped-resources/operators.coreos.com/operators/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.259573502Z cluster-scoped-resources/operators.coreos.com/operators/authorino-operator.kuadrant-system.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.259714628Z cluster-scoped-resources/operators.coreos.com/operators/dns-operator.kuadrant-system.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.259818843Z cluster-scoped-resources/operators.coreos.com/operators/leader-worker-set.openshift-lws-operator.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.260035463Z cluster-scoped-resources/operators.coreos.com/operators/limitador-operator.kuadrant-system.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.260206582Z cluster-scoped-resources/operators.coreos.com/operators/openshift-cert-manager-operator.cert-manager-operator.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.260634592Z cluster-scoped-resources/operators.coreos.com/operators/openshift-custom-metrics-autoscaler-operator.openshift-keda.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.260766598Z cluster-scoped-resources/operators.coreos.com/operators/rhcl-operator.kuadrant-system.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.260920345Z cluster-scoped-resources/rbac.authorization.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.2610235Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.261068772Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/aws-ebs-csi-driver-operator-clusterrolebinding.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.261177417Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/cloud-network-config-controller.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.261303973Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/cluster-node-tuning-operator.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.261502123Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/cluster-storage-operator-role.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.261635229Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/csi-snapshot-controller-operator-clusterrole.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.261723683Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/csi-snapshot-controller-runner-operator.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.261819927Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/metrics-daemon-sa-rolebinding.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.261904611Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-admission-controller-webhook.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.261992966Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-ancillary-tools.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.262114711Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-cluster-readers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.262202745Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-group.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.26229347Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-transient.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.262395834Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-whereabouts.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.262473318Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/network-diagnostics.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.262552362Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/network-node-identity.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.262641506Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-image-registry-pruner.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.2627299Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-iptables-alerter.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.262816074Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.262904129Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-ovn-kubernetes-node-identity-limited.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.263131979Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-ovn-kubernetes-node-kube-rbac-proxy.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.263387221Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/registry-registry-role.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.263502546Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.263551809Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/aws-ebs-csi-driver-operator-clusterrole.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.263664554Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/cloud-network-config-controller.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.263745358Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/cluster-node-tuning-operator.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.263837222Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/console-extensions-reader.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.26399792Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/console-operator.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.264163888Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/console.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.264257502Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/csi-snapshot-controller-operator-clusterrole.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.264344656Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/helm-chartrepos-viewer.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.26443074Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/metrics-daemon-role.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.264518184Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus-admission-controller-webhook.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.264613029Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus-ancillary-tools.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.264705343Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.264792057Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/net-attach-def-project.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.264883122Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/network-diagnostics.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.264985566Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/network-node-identity.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.265124073Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-csi-snapshot-controller-runner.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.26527036Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-iptables-alerter.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.265391606Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-cluster-reader.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.265610156Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.265860537Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-kube-rbac-proxy.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.265951982Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-node-limited.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.266081608Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-udn-editor.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.266193043Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-udn-viewer.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.266291978Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/project-helm-chartrepository-editor.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.266394313Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-admin.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.266490427Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-basic-user.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.266597562Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-storage-admin.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.266733118Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-view.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.266824363Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:registry.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.266942288Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/whereabouts-cni.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.267027683Z cluster-scoped-resources/samples.operator.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.267048733Z cluster-scoped-resources/samples.operator.openshift.io/configs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.267116707Z cluster-scoped-resources/samples.operator.openshift.io/configs/cluster.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.26718725Z cluster-scoped-resources/snapshot.storage.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.267208871Z cluster-scoped-resources/snapshot.storage.k8s.io/volumesnapshotclasses/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.267260563Z cluster-scoped-resources/snapshot.storage.k8s.io/volumesnapshotclasses/csi-aws-vsc.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.267356838Z cluster-scoped-resources/storage.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.267379009Z cluster-scoped-resources/storage.k8s.io/csidrivers/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.267448802Z cluster-scoped-resources/storage.k8s.io/csidrivers/ebs.csi.aws.com.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.26760922Z cluster-scoped-resources/storage.k8s.io/csinodes/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.267809109Z cluster-scoped-resources/storage.k8s.io/csinodes/ip-10-0-129-80.ec2.internal.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.267934285Z cluster-scoped-resources/storage.k8s.io/csinodes/ip-10-0-133-86.ec2.internal.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.268064221Z cluster-scoped-resources/storage.k8s.io/csinodes/ip-10-0-140-240.ec2.internal.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.268126554Z cluster-scoped-resources/storage.k8s.io/storageclasses/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.268185917Z cluster-scoped-resources/storage.k8s.io/storageclasses/gp2-csi.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.268297822Z cluster-scoped-resources/storage.k8s.io/storageclasses/gp3-csi.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.268381296Z cluster-scoped-resources/storage.k8s.io/volumeattachments/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.268444289Z cluster-scoped-resources/storage.k8s.io/volumeattachments/csi-073bc2b8f16dfea97aee88d62910a618d9c2b6da144d4a74033e2cdbcfbe1d32.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.268573285Z cluster-scoped-resources/storage.k8s.io/volumeattachments/csi-8ff94c8e73e106c66a59b8f82c20c6111f6926f6cfa82df1e31b0a6ce37431b1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.268703001Z host_service_logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.268743523Z host_service_logs/masters/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.268807486Z host_service_logs/masters/NetworkManager_service.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.26888948Z host_service_logs/masters/crio_service.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.268980814Z host_service_logs/masters/kubelet_service.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.26909447Z host_service_logs/masters/machine-config-daemon-firstboot_service.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.269260757Z host_service_logs/masters/machine-config-daemon-host_service.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.269409344Z host_service_logs/masters/openvswitch_service.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.269502699Z host_service_logs/masters/ostree-finalize-staged_service.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.269609324Z host_service_logs/masters/ovs-configuration_service.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.269707689Z host_service_logs/masters/ovs-vswitchd_service.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.269805083Z host_service_logs/masters/ovsdb-server_service.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.269891477Z host_service_logs/masters/rpm-ostreed_service.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.26995746Z ingress_controllers/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.269967651Z ingress_controllers/default/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.269971791Z ingress_controllers/default/router-default-d6d96c7dc-qc8qb/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.270054365Z ingress_controllers/default/router-default-d6d96c7dc-qc8qb/haproxy.config [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.270252104Z insights-data/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.270287876Z istio/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.270361929Z istio/aggregated-discovery-api.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.270510376Z istio/aggregated-discovery-apis.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.27123927Z istio/event-filter.html [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.271789387Z istio/timestamp [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.271900952Z istio/cluster-scoped-resources/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.271911812Z istio/cluster-scoped-resources/admissionregistration.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.271917582Z istio/cluster-scoped-resources/admissionregistration.k8s.io/mutatingwebhookconfigurations/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.271939034Z istio/cluster-scoped-resources/admissionregistration.k8s.io/mutatingwebhookconfigurations/istio-sidecar-injector-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.272117872Z istio/cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.272167024Z istio/cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/istio-validator-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.272232077Z istio/cluster-scoped-resources/apiextensions.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.27228971Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.272345533Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authorizationpolicies.security.istio.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.272514861Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/destinationrules.networking.istio.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.273385582Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/envoyfilters.networking.istio.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.273679295Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gatewayclasses.gateway.networking.k8s.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.274022652Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.gateway.networking.k8s.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.274516515Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.networking.istio.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.274717184Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/grpcroutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.275185226Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/httproutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.276101979Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.k8s.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.276260267Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.276389703Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/peerauthentications.security.istio.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.276560081Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/proxyconfigs.networking.istio.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.276692577Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/referencegrants.gateway.networking.k8s.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.276931818Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/requestauthentications.security.istio.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.277284425Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/serviceentries.networking.istio.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.277526276Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/sidecars.networking.istio.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.277871913Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/telemetries.telemetry.istio.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.278117784Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/virtualservices.networking.istio.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.278526913Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/wasmplugins.extensions.istio.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.27866765Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadentries.networking.istio.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.278823527Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadgroups.networking.istio.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.279032827Z istio/cluster-scoped-resources/gateway.networking.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.279058818Z istio/cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.279079759Z istio/cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/openshift-default.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.279147733Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.279157453Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.279170184Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/istio-reader-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.27929595Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/istiod-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.279459057Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/istiod-gateway-controller-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.279557032Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.279661637Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/istio-reader-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.279784653Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/istiod-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.279977682Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/istiod-gateway-controller-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.280173631Z istio/namespaces/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.280187042Z istio/namespaces/e2e-test-llm-inference-service-48639af5/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.280195822Z istio/namespaces/e2e-test-llm-inference-service-48639af5/gateway.networking.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.280203932Z istio/namespaces/e2e-test-llm-inference-service-48639af5/gateway.networking.k8s.io/httproutes/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.280254875Z istio/namespaces/e2e-test-llm-inference-service-48639af5/gateway.networking.k8s.io/httproutes/llmisvc-model-pvc-router-manage-e8706282-kserve-route.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.280398512Z istio/namespaces/e2e-test-llm-inference-service-48639af5/inference.networking.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.280407742Z istio/namespaces/e2e-test-llm-inference-service-48639af5/inference.networking.k8s.io/inferencepools/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.280447944Z istio/namespaces/e2e-test-llm-inference-service-48639af5/inference.networking.k8s.io/inferencepools/llmisvc-model-pvc-router-manage-e8706282-inference-pool.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.280512247Z istio/namespaces/e2e-test-llm-inference-service-48639af5/inference.networking.x-k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.280529518Z istio/namespaces/e2e-test-llm-inference-service-48639af5/inference.networking.x-k8s.io/inferencepools/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.280537918Z istio/namespaces/e2e-test-llm-inference-service-48639af5/inference.networking.x-k8s.io/inferencepools/llmisvc-model-pvc-router-manage-e8706282-inference-pool.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.280618962Z istio/namespaces/e2e-test-llm-inference-service-48639af5/networking.istio.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.280685165Z istio/namespaces/e2e-test-llm-inference-service-48639af5/networking.istio.io/destinationrules/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.280749368Z istio/namespaces/e2e-test-llm-inference-service-48639af5/networking.istio.io/destinationrules/llmisvc-model-pvc-router-manage-e8706282-kserve-scheduler.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.280863413Z istio/namespaces/e2e-test-llm-inference-service-48639af5/networking.istio.io/destinationrules/llmisvc-model-pvc-router-manage-e8706282-kserve-shadow-svc.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.281108195Z istio/namespaces/e2e-test-llm-inference-service-48639af5/networking.istio.io/destinationrules/llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.281230111Z istio/namespaces/e2e-test-llm-inference-service-7c3633d1/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.281238241Z istio/namespaces/e2e-test-llm-inference-service-7c3633d1/gateway.networking.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.281245031Z istio/namespaces/e2e-test-llm-inference-service-7c3633d1/gateway.networking.k8s.io/httproutes/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.281310964Z istio/namespaces/e2e-test-llm-inference-service-7c3633d1/gateway.networking.k8s.io/httproutes/llmisvc-model-pvc-router-manage-2577e794-kserve-route.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.281407529Z istio/namespaces/e2e-test-llm-inference-service-7c3633d1/inference.networking.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.281414859Z istio/namespaces/e2e-test-llm-inference-service-7c3633d1/inference.networking.k8s.io/inferencepools/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.281465392Z istio/namespaces/e2e-test-llm-inference-service-7c3633d1/inference.networking.k8s.io/inferencepools/llmisvc-model-pvc-router-manage-2577e794-inference-pool.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.281545375Z istio/namespaces/e2e-test-llm-inference-service-7c3633d1/inference.networking.x-k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.281552586Z istio/namespaces/e2e-test-llm-inference-service-7c3633d1/inference.networking.x-k8s.io/inferencepools/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.281612509Z istio/namespaces/e2e-test-llm-inference-service-7c3633d1/inference.networking.x-k8s.io/inferencepools/llmisvc-model-pvc-router-manage-2577e794-inference-pool.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.281692373Z istio/namespaces/e2e-test-llm-inference-service-7c3633d1/networking.istio.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.281701203Z istio/namespaces/e2e-test-llm-inference-service-7c3633d1/networking.istio.io/destinationrules/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.281762236Z istio/namespaces/e2e-test-llm-inference-service-7c3633d1/networking.istio.io/destinationrules/llmisvc-model-pvc-router-manage-2577e794-kserve-scheduler.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.281879011Z istio/namespaces/e2e-test-llm-inference-service-7c3633d1/networking.istio.io/destinationrules/llmisvc-model-pvc-router-manage-2577e794-kserve-shadow-svc.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.28206Z istio/namespaces/e2e-test-llm-inference-service-7c3633d1/networking.istio.io/destinationrules/llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.282202297Z istio/namespaces/openshift-ingress/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.282246979Z istio/namespaces/openshift-ingress/openshift-ingress.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.282308121Z istio/namespaces/openshift-ingress/apps.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.282346693Z istio/namespaces/openshift-ingress/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.282422317Z istio/namespaces/openshift-ingress/apps/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.282470529Z istio/namespaces/openshift-ingress/apps/daemonsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.282561743Z istio/namespaces/openshift-ingress/apps/deployments.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.282816805Z istio/namespaces/openshift-ingress/apps/replicasets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.283318969Z istio/namespaces/openshift-ingress/apps/statefulsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.283366861Z istio/namespaces/openshift-ingress/autoscaling/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.283399923Z istio/namespaces/openshift-ingress/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.283486607Z istio/namespaces/openshift-ingress/batch/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.283536129Z istio/namespaces/openshift-ingress/batch/cronjobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.283624873Z istio/namespaces/openshift-ingress/batch/jobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.283667815Z istio/namespaces/openshift-ingress/build.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.283710467Z istio/namespaces/openshift-ingress/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.283792451Z istio/namespaces/openshift-ingress/build.openshift.io/builds.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.283878565Z istio/namespaces/openshift-ingress/core/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.283914007Z istio/namespaces/openshift-ingress/core/configmaps.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.284428691Z istio/namespaces/openshift-ingress/core/endpoints.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.284533906Z istio/namespaces/openshift-ingress/core/events.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.284756257Z istio/namespaces/openshift-ingress/core/persistentvolumeclaims.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.284841001Z istio/namespaces/openshift-ingress/core/pods.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.285128604Z istio/namespaces/openshift-ingress/core/replicationcontrollers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.285270451Z istio/namespaces/openshift-ingress/core/secrets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.28547768Z istio/namespaces/openshift-ingress/core/services.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.285656749Z istio/namespaces/openshift-ingress/discovery.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.285787245Z istio/namespaces/openshift-ingress/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.28589983Z istio/namespaces/openshift-ingress/gateway.networking.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.285911791Z istio/namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.285963863Z istio/namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/openshift-ai-inference.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.286083889Z istio/namespaces/openshift-ingress/image.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.28610735Z istio/namespaces/openshift-ingress/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.286185934Z istio/namespaces/openshift-ingress/k8s.ovn.org/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.286231636Z istio/namespaces/openshift-ingress/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.28631997Z istio/namespaces/openshift-ingress/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.286405734Z istio/namespaces/openshift-ingress/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.286469187Z istio/namespaces/openshift-ingress/monitoring.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.2865187Z istio/namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.286589693Z istio/namespaces/openshift-ingress/networking.istio.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.286599553Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.286647815Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/kuadrant-auth-openshift-ai-inference.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.28674916Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/kuadrant-openshift-ai-inference.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.286898537Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/openshift-ai-inference-authn-ssl.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.287023333Z istio/namespaces/openshift-ingress/networking.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.287083496Z istio/namespaces/openshift-ingress/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.287145169Z istio/namespaces/openshift-ingress/pods/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.287154479Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-46d7p/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.287209172Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-46d7p/istiod-openshift-gateway-5677ccb49b-46d7p.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.287354699Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-46d7p/discovery/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.287362639Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-46d7p/discovery/discovery/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.2873694Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-46d7p/discovery/discovery/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.287424142Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-46d7p/discovery/discovery/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.298542705Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-46d7p/discovery/discovery/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.298618029Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-46d7p/discovery/discovery/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.29865772Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-xf9js/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.298709383Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-xf9js/openshift-ai-inference-openshift-default-8699b59bbc-xf9js.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.298827119Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-xf9js/istio-proxy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.298846289Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-xf9js/istio-proxy/istio-proxy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.29885281Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-xf9js/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.298902912Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-xf9js/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.300424984Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-xf9js/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.300509618Z istio/namespaces/openshift-ingress/pods/router-default-d6d96c7dc-qc8qb/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.30055997Z istio/namespaces/openshift-ingress/pods/router-default-d6d96c7dc-qc8qb/router-default-d6d96c7dc-qc8qb.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.300669985Z istio/namespaces/openshift-ingress/pods/router-default-d6d96c7dc-qc8qb/router/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.300678906Z istio/namespaces/openshift-ingress/pods/router-default-d6d96c7dc-qc8qb/router/router/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.300683816Z istio/namespaces/openshift-ingress/pods/router-default-d6d96c7dc-qc8qb/router/router/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.300708357Z istio/namespaces/openshift-ingress/pods/router-default-d6d96c7dc-qc8qb/router/router/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.300855214Z istio/namespaces/openshift-ingress/pods/router-default-d6d96c7dc-qc8qb/router/router/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.300927427Z istio/namespaces/openshift-ingress/pods/router-default-d6d96c7dc-qc8qb/router/router/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.300956699Z istio/namespaces/openshift-ingress/policy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.301019602Z istio/namespaces/openshift-ingress/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.301088915Z istio/namespaces/openshift-ingress/route.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.301124216Z istio/namespaces/openshift-ingress/route.openshift.io/routes.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.301177599Z machine_config_ondisk/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.301187879Z machine_config_termination_logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.301193Z monitoring/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.30119834Z monitoring/alertmanager/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.301229032Z monitoring/alertmanager/status.json [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.301316826Z monitoring/alertmanager/status.stderr [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.301368678Z monitoring/prometheus/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.301424971Z monitoring/prometheus/alertmanagers.json [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.301513675Z monitoring/prometheus/alertmanagers.stderr [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.301581108Z monitoring/prometheus/rules.json [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.30226112Z monitoring/prometheus/rules.stderr [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.302302852Z monitoring/prometheus/prometheus-k8s-0/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.302351774Z monitoring/prometheus/prometheus-k8s-0/active-targets.json [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.303403634Z monitoring/prometheus/prometheus-k8s-0/active-targets.stderr [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.303463657Z monitoring/prometheus/prometheus-k8s-0/status/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.303509969Z monitoring/prometheus/prometheus-k8s-0/status/runtimeinfo.json [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.303598553Z monitoring/prometheus/prometheus-k8s-0/status/runtimeinfo.stderr [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.303668476Z monitoring/prometheus/prometheus-k8s-0/status/tsdb.json [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.303765471Z monitoring/prometheus/prometheus-k8s-0/status/tsdb.stderr [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.303854225Z monitoring/prometheus/status/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.303902117Z monitoring/prometheus/status/config.json [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.304483295Z monitoring/prometheus/status/config.stderr [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.304540908Z monitoring/prometheus/status/flags.json [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.304639422Z monitoring/prometheus/status/flags.stderr [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.304895254Z namespaces/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.304908665Z namespaces/cert-manager-operator/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.304912975Z namespaces/cert-manager-operator/coordination.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.304919105Z namespaces/cert-manager-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.304983798Z namespaces/cert-manager-operator/coordination.k8s.io/leases/cert-manager-operator-lock.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.305104604Z namespaces/cert-manager-operator/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.305120465Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.305155526Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.305381937Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.305600157Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.305748054Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.305991476Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.306334512Z namespaces/cert-manager-operator/operators.coreos.com/installplans/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.306383914Z namespaces/cert-manager-operator/operators.coreos.com/installplans/install-4gctz.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.306540561Z namespaces/cert-manager-operator/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.306588174Z namespaces/cert-manager-operator/operators.coreos.com/operatorconditions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.306642356Z namespaces/cert-manager-operator/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.306693198Z namespaces/cert-manager-operator/operators.coreos.com/operatorgroups/openshift-cert-manager-operator.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.306751671Z namespaces/cert-manager-operator/operators.coreos.com/subscriptions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.306793743Z namespaces/cert-manager-operator/operators.coreos.com/subscriptions/openshift-cert-manager-operator.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.306873717Z namespaces/cert-manager/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.306883887Z namespaces/cert-manager/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.306893468Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.306977292Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.307189652Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.307396612Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.307546879Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.307749828Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.308027191Z namespaces/default/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.308098945Z namespaces/default/default.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.308176018Z namespaces/default/apps.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.308224481Z namespaces/default/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.308297044Z namespaces/default/apps/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.308346927Z namespaces/default/apps/daemonsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.308469852Z namespaces/default/apps/deployments.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.308684292Z namespaces/default/apps/replicasets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.30884987Z namespaces/default/apps/statefulsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.308946115Z namespaces/default/autoscaling/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.309019158Z namespaces/default/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.309119013Z namespaces/default/batch/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.309165745Z namespaces/default/batch/cronjobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.309254959Z namespaces/default/batch/jobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.309320222Z namespaces/default/build.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.309375975Z namespaces/default/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.309470329Z namespaces/default/build.openshift.io/builds.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.309521852Z namespaces/default/core/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.309580065Z namespaces/default/core/configmaps.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.30969356Z namespaces/default/core/endpoints.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.309780134Z namespaces/default/core/events.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.309971493Z namespaces/default/core/persistentvolumeclaims.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.310089698Z namespaces/default/core/pods.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.310178783Z namespaces/default/core/replicationcontrollers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.310323849Z namespaces/default/core/secrets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.310510548Z namespaces/default/core/services.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.310600513Z namespaces/default/discovery.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.310648775Z namespaces/default/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.310731349Z namespaces/default/image.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.31076215Z namespaces/default/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.310826693Z namespaces/default/k8s.ovn.org/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.310872425Z namespaces/default/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.31097315Z namespaces/default/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.311080915Z namespaces/default/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.311133247Z namespaces/default/monitoring.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.31118319Z namespaces/default/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.311255123Z namespaces/default/networking.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.311322236Z namespaces/default/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.311387379Z namespaces/default/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.31139764Z namespaces/default/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.311456573Z namespaces/default/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.311629931Z namespaces/default/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.31183042Z namespaces/default/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.311991458Z namespaces/default/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.31225287Z namespaces/default/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.312479351Z namespaces/default/policy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.312534044Z namespaces/default/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.312597227Z namespaces/default/route.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.312649969Z namespaces/default/route.openshift.io/routes.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.312693431Z namespaces/e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.312702261Z namespaces/e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.312710122Z namespaces/e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.312770664Z namespaces/e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.312966124Z namespaces/e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.313340471Z namespaces/e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.313480778Z namespaces/e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.313865956Z namespaces/e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.314086557Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.314099247Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.314104267Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.31415844Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.314307387Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.314489905Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.314629492Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.314835572Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.315067433Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.315084173Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.315093094Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.315142276Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.315297734Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.315481872Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.315621429Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.315818808Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.316019057Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.316038298Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.316052549Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.316169175Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.316316241Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.31649185Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.316629466Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.316847896Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.317063337Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.317077627Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.317089698Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.31714307Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.31735686Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.317539159Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.317754989Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.317971029Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.318288414Z namespaces/e2e-test-llm-inference-service-48639af5/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.318300205Z namespaces/e2e-test-llm-inference-service-48639af5/monitoring.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.318307015Z namespaces/e2e-test-llm-inference-service-48639af5/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.318355367Z namespaces/e2e-test-llm-inference-service-48639af5/monitoring.coreos.com/podmonitors/kserve-llm-isvc-vllm-engine-default.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.318455592Z namespaces/e2e-test-llm-inference-service-48639af5/monitoring.coreos.com/podmonitors/kserve-llm-isvc-vllm-engine.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.318518775Z namespaces/e2e-test-llm-inference-service-48639af5/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.318564017Z namespaces/e2e-test-llm-inference-service-48639af5/monitoring.coreos.com/servicemonitors/kserve-llm-isvc-scheduler-default.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.318652631Z namespaces/e2e-test-llm-inference-service-48639af5/monitoring.coreos.com/servicemonitors/kserve-llm-isvc-scheduler.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.318713164Z namespaces/e2e-test-llm-inference-service-48639af5/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.318731445Z namespaces/e2e-test-llm-inference-service-48639af5/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.318778337Z namespaces/e2e-test-llm-inference-service-48639af5/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.318943465Z namespaces/e2e-test-llm-inference-service-48639af5/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.319157185Z namespaces/e2e-test-llm-inference-service-48639af5/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.319299382Z namespaces/e2e-test-llm-inference-service-48639af5/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.319498251Z namespaces/e2e-test-llm-inference-service-48639af5/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.31968236Z namespaces/e2e-test-llm-inference-service-7c3633d1/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.31969021Z namespaces/e2e-test-llm-inference-service-7c3633d1/monitoring.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.31969422Z namespaces/e2e-test-llm-inference-service-7c3633d1/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.319727282Z namespaces/e2e-test-llm-inference-service-7c3633d1/monitoring.coreos.com/podmonitors/kserve-llm-isvc-vllm-engine-default.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.319818086Z namespaces/e2e-test-llm-inference-service-7c3633d1/monitoring.coreos.com/podmonitors/kserve-llm-isvc-vllm-engine.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.319867158Z namespaces/e2e-test-llm-inference-service-7c3633d1/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.319910311Z namespaces/e2e-test-llm-inference-service-7c3633d1/monitoring.coreos.com/servicemonitors/kserve-llm-isvc-scheduler-default.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.319993194Z namespaces/e2e-test-llm-inference-service-7c3633d1/monitoring.coreos.com/servicemonitors/kserve-llm-isvc-scheduler.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.320068158Z namespaces/e2e-test-llm-inference-service-7c3633d1/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.320084619Z namespaces/e2e-test-llm-inference-service-7c3633d1/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.320137551Z namespaces/e2e-test-llm-inference-service-7c3633d1/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.320298949Z namespaces/e2e-test-llm-inference-service-7c3633d1/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.320476637Z namespaces/e2e-test-llm-inference-service-7c3633d1/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.320610213Z namespaces/e2e-test-llm-inference-service-7c3633d1/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.320813223Z namespaces/e2e-test-llm-inference-service-7c3633d1/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.320997262Z namespaces/kserve-ci-e2e-test/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.321017363Z namespaces/kserve-ci-e2e-test/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.321048564Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.321093686Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.321249823Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.321440692Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.321576529Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.321890564Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.322308693Z namespaces/kserve/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.322424029Z namespaces/kserve/kserve.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.322504833Z namespaces/kserve/apps.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.322555395Z namespaces/kserve/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.322625188Z namespaces/kserve/apps/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.322672241Z namespaces/kserve/apps/daemonsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.322759765Z namespaces/kserve/apps/deployments.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.322986145Z namespaces/kserve/apps/replicasets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.323204406Z namespaces/kserve/apps/statefulsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.323242467Z namespaces/kserve/autoscaling/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.32329699Z namespaces/kserve/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.323363993Z namespaces/kserve/batch/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.323399985Z namespaces/kserve/batch/cronjobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.323485839Z namespaces/kserve/batch/jobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.323581423Z namespaces/kserve/build.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.323626315Z namespaces/kserve/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.323710969Z namespaces/kserve/build.openshift.io/builds.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.323741291Z namespaces/kserve/coordination.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.323747841Z namespaces/kserve/coordination.k8s.io/leases/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.323800923Z namespaces/kserve/coordination.k8s.io/leases/kserve-controller-manager-leader-lock.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.323891758Z namespaces/kserve/coordination.k8s.io/leases/llminferenceservice-kserve-controller-manager.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.323977782Z namespaces/kserve/coordination.k8s.io/leases/odh-model-controller.opendatahub.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.324094597Z namespaces/kserve/core/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.324131499Z namespaces/kserve/core/configmaps.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.324546779Z namespaces/kserve/core/endpoints.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.324651094Z namespaces/kserve/core/events.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.32499435Z namespaces/kserve/core/persistentvolumeclaims.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.325122106Z namespaces/kserve/core/pods.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.32542255Z namespaces/kserve/core/replicationcontrollers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.325555696Z namespaces/kserve/core/secrets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.325803178Z namespaces/kserve/core/services.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.325947644Z namespaces/kserve/discovery.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.326014218Z namespaces/kserve/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.326194426Z namespaces/kserve/image.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.3262824Z namespaces/kserve/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.326406646Z namespaces/kserve/k8s.ovn.org/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.326453968Z namespaces/kserve/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.326547073Z namespaces/kserve/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.326626727Z namespaces/kserve/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.326671638Z namespaces/kserve/monitoring.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.3267072Z namespaces/kserve/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.326825576Z namespaces/kserve/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.326860007Z namespaces/kserve/monitoring.coreos.com/servicemonitors/model-serving-api-metrics.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.326957722Z namespaces/kserve/monitoring.coreos.com/servicemonitors/odh-model-controller-metrics-monitor.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.327026865Z namespaces/kserve/networking.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.327079338Z namespaces/kserve/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.32713758Z namespaces/kserve/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.327153751Z namespaces/kserve/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.327201594Z namespaces/kserve/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.327379572Z namespaces/kserve/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.327570631Z namespaces/kserve/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.327706257Z namespaces/kserve/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.327912917Z namespaces/kserve/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.328120397Z namespaces/kserve/pods/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.328132077Z namespaces/kserve/pods/kserve-controller-manager-65f566c74b-m6f8j/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.32818342Z namespaces/kserve/pods/kserve-controller-manager-65f566c74b-m6f8j/kserve-controller-manager-65f566c74b-m6f8j.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.328279644Z namespaces/kserve/pods/kserve-controller-manager-65f566c74b-m6f8j/manager/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.328287955Z namespaces/kserve/pods/kserve-controller-manager-65f566c74b-m6f8j/manager/manager/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.328297625Z namespaces/kserve/pods/kserve-controller-manager-65f566c74b-m6f8j/manager/manager/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.328340607Z namespaces/kserve/pods/kserve-controller-manager-65f566c74b-m6f8j/manager/manager/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.328468813Z namespaces/kserve/pods/kserve-controller-manager-65f566c74b-m6f8j/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.328538097Z namespaces/kserve/pods/kserve-controller-manager-65f566c74b-m6f8j/manager/manager/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.328592199Z namespaces/kserve/pods/llmisvc-controller-manager-84977dd55d-zfng5/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.328636131Z namespaces/kserve/pods/llmisvc-controller-manager-84977dd55d-zfng5/llmisvc-controller-manager-84977dd55d-zfng5.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.328728155Z namespaces/kserve/pods/llmisvc-controller-manager-84977dd55d-zfng5/manager/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.328735306Z namespaces/kserve/pods/llmisvc-controller-manager-84977dd55d-zfng5/manager/manager/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.328744536Z namespaces/kserve/pods/llmisvc-controller-manager-84977dd55d-zfng5/manager/manager/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.328794578Z namespaces/kserve/pods/llmisvc-controller-manager-84977dd55d-zfng5/manager/manager/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.333785703Z namespaces/kserve/pods/llmisvc-controller-manager-84977dd55d-zfng5/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.333852776Z namespaces/kserve/pods/llmisvc-controller-manager-84977dd55d-zfng5/manager/manager/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.333905419Z namespaces/kserve/pods/model-serving-api-7b954ffc95-bj279/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.333940031Z namespaces/kserve/pods/model-serving-api-7b954ffc95-bj279/model-serving-api-7b954ffc95-bj279.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.334064356Z namespaces/kserve/pods/model-serving-api-7b954ffc95-bj279/server/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.334076197Z namespaces/kserve/pods/model-serving-api-7b954ffc95-bj279/server/server/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.334082107Z namespaces/kserve/pods/model-serving-api-7b954ffc95-bj279/server/server/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.334124859Z namespaces/kserve/pods/model-serving-api-7b954ffc95-bj279/server/server/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.334217953Z namespaces/kserve/pods/model-serving-api-7b954ffc95-bj279/server/server/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.334287767Z namespaces/kserve/pods/model-serving-api-7b954ffc95-bj279/server/server/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.334324198Z namespaces/kserve/pods/odh-model-controller-75d9cdbddf-6h4sv/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.334377981Z namespaces/kserve/pods/odh-model-controller-75d9cdbddf-6h4sv/odh-model-controller-75d9cdbddf-6h4sv.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.334478196Z namespaces/kserve/pods/odh-model-controller-75d9cdbddf-6h4sv/manager/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.334486956Z namespaces/kserve/pods/odh-model-controller-75d9cdbddf-6h4sv/manager/manager/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.334494897Z namespaces/kserve/pods/odh-model-controller-75d9cdbddf-6h4sv/manager/manager/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.334550479Z namespaces/kserve/pods/odh-model-controller-75d9cdbddf-6h4sv/manager/manager/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.367644606Z namespaces/kserve/pods/odh-model-controller-75d9cdbddf-6h4sv/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.367689768Z namespaces/kserve/pods/odh-model-controller-75d9cdbddf-6h4sv/manager/manager/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.367704469Z namespaces/kserve/pods/s3-init-bg8rw/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.367742801Z namespaces/kserve/pods/s3-init-bg8rw/s3-init-bg8rw.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.367881108Z namespaces/kserve/pods/s3-init-bg8rw/download-hf-model/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.367893348Z namespaces/kserve/pods/s3-init-bg8rw/download-hf-model/download-hf-model/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.367898918Z namespaces/kserve/pods/s3-init-bg8rw/download-hf-model/download-hf-model/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.367908569Z namespaces/kserve/pods/s3-init-bg8rw/download-hf-model/download-hf-model/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.368025614Z namespaces/kserve/pods/s3-init-bg8rw/download-hf-model/download-hf-model/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.368099358Z namespaces/kserve/pods/s3-init-bg8rw/download-hf-model/download-hf-model/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.368116008Z namespaces/kserve/pods/s3-init-bg8rw/s3-init/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.368132449Z namespaces/kserve/pods/s3-init-bg8rw/s3-init/s3-init/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.36814702Z namespaces/kserve/pods/s3-init-bg8rw/s3-init/s3-init/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.368184572Z namespaces/kserve/pods/s3-init-bg8rw/s3-init/s3-init/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.368459495Z namespaces/kserve/pods/s3-init-bg8rw/s3-init/s3-init/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.368528448Z namespaces/kserve/pods/s3-init-bg8rw/s3-init/s3-init/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.368548689Z namespaces/kserve/pods/seaweedfs-64568bcd49-r6dnp/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.368602071Z namespaces/kserve/pods/seaweedfs-64568bcd49-r6dnp/seaweedfs-64568bcd49-r6dnp.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.368681725Z namespaces/kserve/pods/seaweedfs-64568bcd49-r6dnp/seaweedfs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.368690946Z namespaces/kserve/pods/seaweedfs-64568bcd49-r6dnp/seaweedfs/seaweedfs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.368702976Z namespaces/kserve/pods/seaweedfs-64568bcd49-r6dnp/seaweedfs/seaweedfs/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.368739578Z namespaces/kserve/pods/seaweedfs-64568bcd49-r6dnp/seaweedfs/seaweedfs/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.368949858Z namespaces/kserve/pods/seaweedfs-64568bcd49-r6dnp/seaweedfs/seaweedfs/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.369033772Z namespaces/kserve/pods/seaweedfs-64568bcd49-r6dnp/seaweedfs/seaweedfs/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.369077154Z namespaces/kserve/policy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.369125076Z namespaces/kserve/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.369173368Z namespaces/kserve/route.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.36920761Z namespaces/kserve/route.openshift.io/routes.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.369302764Z namespaces/kuadrant-system/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.369350737Z namespaces/kuadrant-system/kuadrant-system.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.36942188Z namespaces/kuadrant-system/apps.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.369464562Z namespaces/kuadrant-system/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.369542926Z namespaces/kuadrant-system/apps/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.369599578Z namespaces/kuadrant-system/apps/daemonsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.369687673Z namespaces/kuadrant-system/apps/deployments.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.370001647Z namespaces/kuadrant-system/apps/replicasets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.370297401Z namespaces/kuadrant-system/apps/statefulsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.370352544Z namespaces/kuadrant-system/autoscaling/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.370393336Z namespaces/kuadrant-system/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.370452058Z namespaces/kuadrant-system/batch/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.370509611Z namespaces/kuadrant-system/batch/cronjobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.370593525Z namespaces/kuadrant-system/batch/jobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.370626277Z namespaces/kuadrant-system/build.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.370677109Z namespaces/kuadrant-system/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.370758633Z namespaces/kuadrant-system/build.openshift.io/builds.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.370804255Z namespaces/kuadrant-system/coordination.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.370830896Z namespaces/kuadrant-system/coordination.k8s.io/leases/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.370882239Z namespaces/kuadrant-system/coordination.k8s.io/leases/3745a16e.kuadrant.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.370968543Z namespaces/kuadrant-system/coordination.k8s.io/leases/a3f98d6c.kuadrant.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.371104389Z namespaces/kuadrant-system/coordination.k8s.io/leases/aac3a15d.authorino.kuadrant.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.371303449Z namespaces/kuadrant-system/coordination.k8s.io/leases/f139389e.kuadrant.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.371414194Z namespaces/kuadrant-system/core/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.371444695Z namespaces/kuadrant-system/core/configmaps.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.371619863Z namespaces/kuadrant-system/core/endpoints.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.371735549Z namespaces/kuadrant-system/core/events.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.37218518Z namespaces/kuadrant-system/core/persistentvolumeclaims.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.372274754Z namespaces/kuadrant-system/core/pods.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.37260205Z namespaces/kuadrant-system/core/replicationcontrollers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.372751106Z namespaces/kuadrant-system/core/secrets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.372966247Z namespaces/kuadrant-system/core/services.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.373088702Z namespaces/kuadrant-system/discovery.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.373130515Z namespaces/kuadrant-system/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.37324591Z namespaces/kuadrant-system/image.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.373279671Z namespaces/kuadrant-system/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.373321593Z namespaces/kuadrant-system/k8s.ovn.org/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.373363325Z namespaces/kuadrant-system/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.373443319Z namespaces/kuadrant-system/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.373525303Z namespaces/kuadrant-system/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.373566235Z namespaces/kuadrant-system/monitoring.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.373609777Z namespaces/kuadrant-system/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.373656189Z namespaces/kuadrant-system/networking.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.373695711Z namespaces/kuadrant-system/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.373759294Z namespaces/kuadrant-system/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.373769125Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.373824027Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.374181864Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.374405095Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.374549511Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.374759151Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.374963601Z namespaces/kuadrant-system/operators.coreos.com/installplans/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.375018673Z namespaces/kuadrant-system/operators.coreos.com/installplans/install-xjnmt.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.375413962Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.375461284Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.375654173Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.37579357Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.375905385Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.375956677Z namespaces/kuadrant-system/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.37601734Z namespaces/kuadrant-system/operators.coreos.com/operatorgroups/kuadrant.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.376072673Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.376119565Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/authorino-operator-stable-redhat-operators-openshift-marketplace.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.37621519Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/dns-operator-stable-redhat-operators-openshift-marketplace.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.376371227Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/limitador-operator-stable-redhat-operators-openshift-marketplace.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.376464131Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/rhcl-operator.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.376547095Z namespaces/kuadrant-system/pods/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.376555536Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-5j7sb/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.376579397Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-5j7sb/authorino-7cf4765b85-5j7sb.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.376676321Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-5j7sb/authorino/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.376683692Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-5j7sb/authorino/authorino/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.376691732Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-5j7sb/authorino/authorino/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.376746034Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-5j7sb/authorino/authorino/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.390294132Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-5j7sb/authorino/authorino/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.390367026Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-5j7sb/authorino/authorino/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.390387816Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-rzd9v/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.390442319Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-rzd9v/authorino-operator-56fb6f6856-rzd9v.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.390544654Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-rzd9v/manager/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.390558724Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-rzd9v/manager/manager/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.390568145Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-rzd9v/manager/manager/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.390605547Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-rzd9v/manager/manager/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.390788975Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-rzd9v/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.390854378Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-rzd9v/manager/manager/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.390911681Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-9wzdw/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.390953623Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-9wzdw/dns-operator-controller-manager-ccbdf686-9wzdw.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.391077179Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-9wzdw/manager/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.39109949Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-9wzdw/manager/manager/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.39110974Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-9wzdw/manager/manager/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.391144632Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-9wzdw/manager/manager/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.391251947Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-9wzdw/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.391332151Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-9wzdw/manager/manager/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.391384693Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-gwjp6/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.391431896Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-gwjp6/kuadrant-console-plugin-598f86468b-gwjp6.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.391508779Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-gwjp6/kuadrant-console-plugin/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.391516989Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-gwjp6/kuadrant-console-plugin/kuadrant-console-plugin/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.391529Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-gwjp6/kuadrant-console-plugin/kuadrant-console-plugin/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.391577932Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-gwjp6/kuadrant-console-plugin/kuadrant-console-plugin/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.391654666Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-gwjp6/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.391724289Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-gwjp6/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.391754701Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-m5dpg/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.391783552Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-m5dpg/kuadrant-operator-controller-manager-66866f5fcc-m5dpg.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.391908628Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-m5dpg/manager/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.391915968Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-m5dpg/manager/manager/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.391919999Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-m5dpg/manager/manager/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.39195521Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-m5dpg/manager/manager/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.404778853Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-m5dpg/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.404845196Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-m5dpg/manager/manager/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.404877798Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-c9rqr/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.404934071Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-c9rqr/limitador-limitador-85d9db4d5d-c9rqr.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.405026135Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-c9rqr/limitador/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.405054606Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-c9rqr/limitador/limitador/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.405068277Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-c9rqr/limitador/limitador/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.405092228Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-c9rqr/limitador/limitador/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.405168132Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-c9rqr/limitador/limitador/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.405246385Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-c9rqr/limitador/limitador/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.405293468Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-fdkhh/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.40534352Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-fdkhh/limitador-operator-controller-manager-774d94476f-fdkhh.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.405460785Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-fdkhh/manager/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.405472826Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-fdkhh/manager/manager/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.405477716Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-fdkhh/manager/manager/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.405532689Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-fdkhh/manager/manager/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.405709897Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-fdkhh/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.405791371Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-fdkhh/manager/manager/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.405857924Z namespaces/kuadrant-system/policy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.405905357Z namespaces/kuadrant-system/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.4059777Z namespaces/kuadrant-system/route.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.406035792Z namespaces/kuadrant-system/route.openshift.io/routes.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.406120756Z namespaces/kube-node-lease/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.406129417Z namespaces/kube-node-lease/coordination.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.406137827Z namespaces/kube-node-lease/coordination.k8s.io/leases/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.40619852Z namespaces/kube-node-lease/coordination.k8s.io/leases/ip-10-0-129-80.ec2.internal.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.406312145Z namespaces/kube-node-lease/coordination.k8s.io/leases/ip-10-0-133-86.ec2.internal.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.40641214Z namespaces/kube-node-lease/coordination.k8s.io/leases/ip-10-0-140-240.ec2.internal.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.406460752Z namespaces/kube-node-lease/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.406470133Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.406576128Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.406797839Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.407216208Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.407448119Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.407702541Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.407918881Z namespaces/kube-public/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.407928862Z namespaces/kube-public/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.407939222Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.407988664Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.408159372Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.408339331Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.408545631Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.408882246Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.409141359Z namespaces/kube-system/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.409189011Z namespaces/kube-system/kube-system.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.40960175Z namespaces/kube-system/apps.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.409654943Z namespaces/kube-system/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.409726786Z namespaces/kube-system/apps/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.409760398Z namespaces/kube-system/apps/daemonsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.409942046Z namespaces/kube-system/apps/deployments.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.41002323Z namespaces/kube-system/apps/replicasets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.410128915Z namespaces/kube-system/apps/statefulsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.410183608Z namespaces/kube-system/autoscaling/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.41022936Z namespaces/kube-system/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.410297213Z namespaces/kube-system/batch/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.410342675Z namespaces/kube-system/batch/cronjobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.4104418Z namespaces/kube-system/batch/jobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.410476032Z namespaces/kube-system/build.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.410543515Z namespaces/kube-system/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.41066943Z namespaces/kube-system/build.openshift.io/builds.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.410729503Z namespaces/kube-system/coordination.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.410737814Z namespaces/kube-system/coordination.k8s.io/leases/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.410786586Z namespaces/kube-system/coordination.k8s.io/leases/apiserver-zggsev2tscfv7c6c2bcmk4rd5a.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.410887951Z namespaces/kube-system/coordination.k8s.io/leases/cert-manager-cainjector-leader-election.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.410975075Z namespaces/kube-system/coordination.k8s.io/leases/cert-manager-controller.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.41107678Z namespaces/kube-system/coordination.k8s.io/leases/kube-controller-manager.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.411228347Z namespaces/kube-system/coordination.k8s.io/leases/kube-scheduler.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.411314611Z namespaces/kube-system/core/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.411358803Z namespaces/kube-system/core/configmaps.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.4115151Z namespaces/kube-system/core/endpoints.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.411599124Z namespaces/kube-system/core/events.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.411860227Z namespaces/kube-system/core/persistentvolumeclaims.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.41193806Z namespaces/kube-system/core/pods.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.412304178Z namespaces/kube-system/core/replicationcontrollers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.412434114Z namespaces/kube-system/core/secrets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.41278692Z namespaces/kube-system/core/services.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.412854504Z namespaces/kube-system/discovery.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.412932957Z namespaces/kube-system/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.41299349Z namespaces/kube-system/image.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.413062773Z namespaces/kube-system/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.413123786Z namespaces/kube-system/k8s.ovn.org/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.413161288Z namespaces/kube-system/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.413258772Z namespaces/kube-system/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.413338716Z namespaces/kube-system/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.41342243Z namespaces/kube-system/monitoring.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.413471513Z namespaces/kube-system/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.413524995Z namespaces/kube-system/networking.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.413604729Z namespaces/kube-system/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.413666602Z namespaces/kube-system/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.413674912Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.413717234Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.413870051Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.414081101Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.414221668Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.414413077Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.414637177Z namespaces/kube-system/pods/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.414644228Z namespaces/kube-system/pods/global-pull-secret-syncer-dsndc/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.41469324Z namespaces/kube-system/pods/global-pull-secret-syncer-dsndc/global-pull-secret-syncer-dsndc.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.414794325Z namespaces/kube-system/pods/global-pull-secret-syncer-dsndc/global-pull-secret-syncer/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.414802615Z namespaces/kube-system/pods/global-pull-secret-syncer-dsndc/global-pull-secret-syncer/global-pull-secret-syncer/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.414809165Z namespaces/kube-system/pods/global-pull-secret-syncer-dsndc/global-pull-secret-syncer/global-pull-secret-syncer/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.414856298Z namespaces/kube-system/pods/global-pull-secret-syncer-dsndc/global-pull-secret-syncer/global-pull-secret-syncer/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.415613993Z namespaces/kube-system/pods/global-pull-secret-syncer-dsndc/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.415772641Z namespaces/kube-system/pods/global-pull-secret-syncer-dsndc/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.415855565Z namespaces/kube-system/pods/global-pull-secret-syncer-nh6vt/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.415902127Z namespaces/kube-system/pods/global-pull-secret-syncer-nh6vt/global-pull-secret-syncer-nh6vt.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.41597765Z namespaces/kube-system/pods/global-pull-secret-syncer-nh6vt/global-pull-secret-syncer/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.415988951Z namespaces/kube-system/pods/global-pull-secret-syncer-nh6vt/global-pull-secret-syncer/global-pull-secret-syncer/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.415994151Z namespaces/kube-system/pods/global-pull-secret-syncer-nh6vt/global-pull-secret-syncer/global-pull-secret-syncer/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.416039323Z namespaces/kube-system/pods/global-pull-secret-syncer-nh6vt/global-pull-secret-syncer/global-pull-secret-syncer/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.416412331Z namespaces/kube-system/pods/global-pull-secret-syncer-nh6vt/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.416481484Z namespaces/kube-system/pods/global-pull-secret-syncer-nh6vt/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.416510656Z namespaces/kube-system/pods/global-pull-secret-syncer-r778m/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.416558888Z namespaces/kube-system/pods/global-pull-secret-syncer-r778m/global-pull-secret-syncer-r778m.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.416631741Z namespaces/kube-system/pods/global-pull-secret-syncer-r778m/global-pull-secret-syncer/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.416639002Z namespaces/kube-system/pods/global-pull-secret-syncer-r778m/global-pull-secret-syncer/global-pull-secret-syncer/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.416643252Z namespaces/kube-system/pods/global-pull-secret-syncer-r778m/global-pull-secret-syncer/global-pull-secret-syncer/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.416673613Z namespaces/kube-system/pods/global-pull-secret-syncer-r778m/global-pull-secret-syncer/global-pull-secret-syncer/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.417066912Z namespaces/kube-system/pods/global-pull-secret-syncer-r778m/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.417135345Z namespaces/kube-system/pods/global-pull-secret-syncer-r778m/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.417164806Z namespaces/kube-system/pods/konnectivity-agent-2kzz5/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.417211409Z namespaces/kube-system/pods/konnectivity-agent-2kzz5/konnectivity-agent-2kzz5.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.417290832Z namespaces/kube-system/pods/konnectivity-agent-2kzz5/konnectivity-agent/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.417297802Z namespaces/kube-system/pods/konnectivity-agent-2kzz5/konnectivity-agent/konnectivity-agent/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.417310363Z namespaces/kube-system/pods/konnectivity-agent-2kzz5/konnectivity-agent/konnectivity-agent/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.417353595Z namespaces/kube-system/pods/konnectivity-agent-2kzz5/konnectivity-agent/konnectivity-agent/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.418490538Z namespaces/kube-system/pods/konnectivity-agent-2kzz5/konnectivity-agent/konnectivity-agent/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.418567342Z namespaces/kube-system/pods/konnectivity-agent-2kzz5/konnectivity-agent/konnectivity-agent/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.418605404Z namespaces/kube-system/pods/konnectivity-agent-dsdrb/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.418655376Z namespaces/kube-system/pods/konnectivity-agent-dsdrb/konnectivity-agent-dsdrb.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.41873722Z namespaces/kube-system/pods/konnectivity-agent-dsdrb/konnectivity-agent/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.418745321Z namespaces/kube-system/pods/konnectivity-agent-dsdrb/konnectivity-agent/konnectivity-agent/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.418752041Z namespaces/kube-system/pods/konnectivity-agent-dsdrb/konnectivity-agent/konnectivity-agent/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.418789803Z namespaces/kube-system/pods/konnectivity-agent-dsdrb/konnectivity-agent/konnectivity-agent/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.419963688Z namespaces/kube-system/pods/konnectivity-agent-dsdrb/konnectivity-agent/konnectivity-agent/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.420079023Z namespaces/kube-system/pods/konnectivity-agent-dsdrb/konnectivity-agent/konnectivity-agent/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.420155527Z namespaces/kube-system/pods/konnectivity-agent-zkcxm/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.420208019Z namespaces/kube-system/pods/konnectivity-agent-zkcxm/konnectivity-agent-zkcxm.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.420276122Z namespaces/kube-system/pods/konnectivity-agent-zkcxm/konnectivity-agent/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.420285313Z namespaces/kube-system/pods/konnectivity-agent-zkcxm/konnectivity-agent/konnectivity-agent/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.420290353Z namespaces/kube-system/pods/konnectivity-agent-zkcxm/konnectivity-agent/konnectivity-agent/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.420322195Z namespaces/kube-system/pods/konnectivity-agent-zkcxm/konnectivity-agent/konnectivity-agent/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.421751292Z namespaces/kube-system/pods/konnectivity-agent-zkcxm/konnectivity-agent/konnectivity-agent/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.421823215Z namespaces/kube-system/pods/konnectivity-agent-zkcxm/konnectivity-agent/konnectivity-agent/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.421856717Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-129-80.ec2.internal/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.421888478Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-129-80.ec2.internal/kube-apiserver-proxy-ip-10-0-129-80.ec2.internal.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.421982523Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-129-80.ec2.internal/haproxy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.421993033Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-129-80.ec2.internal/haproxy/haproxy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.421997143Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-129-80.ec2.internal/haproxy/haproxy/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.422079767Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-129-80.ec2.internal/haproxy/haproxy/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.422291648Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-129-80.ec2.internal/haproxy/haproxy/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.422361041Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-129-80.ec2.internal/haproxy/haproxy/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.422413343Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-133-86.ec2.internal/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.422443164Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-133-86.ec2.internal/kube-apiserver-proxy-ip-10-0-133-86.ec2.internal.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.42255331Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-133-86.ec2.internal/haproxy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.42256083Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-133-86.ec2.internal/haproxy/haproxy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.4225648Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-133-86.ec2.internal/haproxy/haproxy/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.422609752Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-133-86.ec2.internal/haproxy/haproxy/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.422898496Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-133-86.ec2.internal/haproxy/haproxy/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.422968689Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-133-86.ec2.internal/haproxy/haproxy/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.423025492Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-140-240.ec2.internal/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.423089905Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-140-240.ec2.internal/kube-apiserver-proxy-ip-10-0-140-240.ec2.internal.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.423160728Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-140-240.ec2.internal/haproxy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.423176429Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-140-240.ec2.internal/haproxy/haproxy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.423180729Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-140-240.ec2.internal/haproxy/haproxy/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.423229432Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-140-240.ec2.internal/haproxy/haproxy/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.423755896Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-140-240.ec2.internal/haproxy/haproxy/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.42383917Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-140-240.ec2.internal/haproxy/haproxy/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.423889943Z namespaces/kube-system/policy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.423938315Z namespaces/kube-system/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.424020869Z namespaces/kube-system/rbac.authorization.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.424035469Z namespaces/kube-system/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.424091302Z namespaces/kube-system/rbac.authorization.k8s.io/rolebindings/csi-snapshot-controller-operator-authentication-reader.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.424197727Z namespaces/kube-system/rbac.authorization.k8s.io/rolebindings/network-diagnostics.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.42426683Z namespaces/kube-system/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.424316213Z namespaces/kube-system/rbac.authorization.k8s.io/roles/extension-apiserver-authentication-reader.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.424386856Z namespaces/kube-system/route.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.424433988Z namespaces/kube-system/route.openshift.io/routes.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.424495431Z namespaces/observability/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.424503352Z namespaces/observability/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.424511952Z namespaces/observability/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.424606646Z namespaces/observability/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.424965133Z namespaces/observability/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.425253767Z namespaces/observability/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.425440145Z namespaces/observability/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.425722509Z namespaces/observability/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.425980571Z namespaces/open-cluster-management-agent-addon/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.425992651Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.426000742Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/leases/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.426080276Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/leases/cluster-proxy.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.426194191Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/leases/work-manager.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.426253264Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.426266415Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.426312347Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.426458693Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.426637962Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.426789389Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.426983088Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.427195608Z namespaces/open-cluster-management-b989455d-6ee2-46a4-81de-c6abc6e08/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.427209009Z namespaces/open-cluster-management-b989455d-6ee2-46a4-81de-c6abc6e08/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.427216559Z namespaces/open-cluster-management-b989455d-6ee2-46a4-81de-c6abc6e08/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.427260291Z namespaces/open-cluster-management-b989455d-6ee2-46a4-81de-c6abc6e08/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.427426939Z namespaces/open-cluster-management-b989455d-6ee2-46a4-81de-c6abc6e08/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.427604187Z namespaces/open-cluster-management-b989455d-6ee2-46a4-81de-c6abc6e08/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.427740084Z namespaces/open-cluster-management-b989455d-6ee2-46a4-81de-c6abc6e08/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.427943363Z namespaces/open-cluster-management-b989455d-6ee2-46a4-81de-c6abc6e08/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.428315771Z namespaces/openshift-apiserver-operator/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.428368483Z namespaces/openshift-apiserver-operator/openshift-apiserver-operator.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.428442187Z namespaces/openshift-apiserver-operator/apps.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.428478719Z namespaces/openshift-apiserver-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.428551622Z namespaces/openshift-apiserver-operator/apps/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.428569623Z namespaces/openshift-apiserver-operator/apps/daemonsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.428670168Z namespaces/openshift-apiserver-operator/apps/deployments.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.428749681Z namespaces/openshift-apiserver-operator/apps/replicasets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.428860917Z namespaces/openshift-apiserver-operator/apps/statefulsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.428901188Z namespaces/openshift-apiserver-operator/autoscaling/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.428953031Z namespaces/openshift-apiserver-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.428995263Z namespaces/openshift-apiserver-operator/batch/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.42915356Z namespaces/openshift-apiserver-operator/batch/cronjobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.429293157Z namespaces/openshift-apiserver-operator/batch/jobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.429337999Z namespaces/openshift-apiserver-operator/build.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.429398062Z namespaces/openshift-apiserver-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.429501707Z namespaces/openshift-apiserver-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.429603311Z namespaces/openshift-apiserver-operator/core/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.429685445Z namespaces/openshift-apiserver-operator/core/configmaps.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.430873411Z namespaces/openshift-apiserver-operator/core/endpoints.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.430961385Z namespaces/openshift-apiserver-operator/core/events.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.43106858Z namespaces/openshift-apiserver-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.431152624Z namespaces/openshift-apiserver-operator/core/pods.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.431235918Z namespaces/openshift-apiserver-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.431373365Z namespaces/openshift-apiserver-operator/core/secrets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.431543873Z namespaces/openshift-apiserver-operator/core/services.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.431612896Z namespaces/openshift-apiserver-operator/discovery.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.431644457Z namespaces/openshift-apiserver-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.43170948Z namespaces/openshift-apiserver-operator/image.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.431739832Z namespaces/openshift-apiserver-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.431804435Z namespaces/openshift-apiserver-operator/k8s.ovn.org/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.431851487Z namespaces/openshift-apiserver-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.431929061Z namespaces/openshift-apiserver-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.432025535Z namespaces/openshift-apiserver-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.432112399Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.432158192Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.432229815Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.432310899Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/servicemonitors/openshift-apiserver-operator.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.432359621Z namespaces/openshift-apiserver-operator/networking.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.432406923Z namespaces/openshift-apiserver-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.432462796Z namespaces/openshift-apiserver-operator/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.432476156Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.432518549Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.432745469Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.432925258Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.433083755Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.433280984Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.433461063Z namespaces/openshift-apiserver-operator/policy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.433489334Z namespaces/openshift-apiserver-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.433546727Z namespaces/openshift-apiserver-operator/route.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.433592339Z namespaces/openshift-apiserver-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.433711995Z namespaces/openshift-apiserver/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.433767317Z namespaces/openshift-apiserver/openshift-apiserver.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.433844451Z namespaces/openshift-apiserver/apps.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.433892683Z namespaces/openshift-apiserver/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.433967347Z namespaces/openshift-apiserver/apps/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.434015039Z namespaces/openshift-apiserver/apps/daemonsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.434136135Z namespaces/openshift-apiserver/apps/deployments.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.434238359Z namespaces/openshift-apiserver/apps/replicasets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.434343515Z namespaces/openshift-apiserver/apps/statefulsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.434397517Z namespaces/openshift-apiserver/autoscaling/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.43445923Z namespaces/openshift-apiserver/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.434520753Z namespaces/openshift-apiserver/batch/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.434586116Z namespaces/openshift-apiserver/batch/cronjobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.43467424Z namespaces/openshift-apiserver/batch/jobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.434742683Z namespaces/openshift-apiserver/build.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.434799536Z namespaces/openshift-apiserver/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.434899941Z namespaces/openshift-apiserver/build.openshift.io/builds.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.434988625Z namespaces/openshift-apiserver/core/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.435064538Z namespaces/openshift-apiserver/core/configmaps.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.435190324Z namespaces/openshift-apiserver/core/endpoints.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.435302529Z namespaces/openshift-apiserver/core/events.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.435401164Z namespaces/openshift-apiserver/core/persistentvolumeclaims.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.435493149Z namespaces/openshift-apiserver/core/pods.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.435596133Z namespaces/openshift-apiserver/core/replicationcontrollers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.435756791Z namespaces/openshift-apiserver/core/secrets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.435964661Z namespaces/openshift-apiserver/core/services.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.436083086Z namespaces/openshift-apiserver/discovery.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.436139159Z namespaces/openshift-apiserver/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.436336788Z namespaces/openshift-apiserver/image.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.436408402Z namespaces/openshift-apiserver/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.436510506Z namespaces/openshift-apiserver/k8s.ovn.org/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.436563479Z namespaces/openshift-apiserver/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.436665954Z namespaces/openshift-apiserver/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.436837702Z namespaces/openshift-apiserver/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.436903415Z namespaces/openshift-apiserver/monitoring.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.436964488Z namespaces/openshift-apiserver/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.437089174Z namespaces/openshift-apiserver/networking.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.437118985Z namespaces/openshift-apiserver/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.437168537Z namespaces/openshift-apiserver/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.437182688Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.437256641Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.43743325Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.43764326Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.437809978Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.438114952Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.438312741Z namespaces/openshift-apiserver/policy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.438354543Z namespaces/openshift-apiserver/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.438418516Z namespaces/openshift-apiserver/route.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.438460008Z namespaces/openshift-apiserver/route.openshift.io/routes.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.438512091Z namespaces/openshift-authentication-operator/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.438519701Z namespaces/openshift-authentication-operator/monitoring.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.438528591Z namespaces/openshift-authentication-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.438569643Z namespaces/openshift-authentication-operator/monitoring.coreos.com/servicemonitors/authentication-operator.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.438640417Z namespaces/openshift-authentication-operator/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.438656367Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.43870635Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.438857267Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.439065157Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.439203063Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.439412143Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.439595602Z namespaces/openshift-authentication/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.439604382Z namespaces/openshift-authentication/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.439617023Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.439665455Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.439821402Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.440018782Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.440180889Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.440390899Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.440701184Z namespaces/openshift-cloud-controller-manager/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.440708614Z namespaces/openshift-cloud-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.440715204Z namespaces/openshift-cloud-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.440854781Z namespaces/openshift-cloud-controller-manager/coordination.k8s.io/leases/cloud-controller-manager.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.440934594Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.440944505Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.440999778Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.441197877Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.441398296Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.441569315Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.441867729Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.44231801Z namespaces/openshift-cloud-credential-operator/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.44232898Z namespaces/openshift-cloud-credential-operator/coordination.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.44233302Z namespaces/openshift-cloud-credential-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.442364472Z namespaces/openshift-cloud-credential-operator/coordination.k8s.io/leases/cloud-credential-operator-leader.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.442440265Z namespaces/openshift-cloud-credential-operator/monitoring.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.442449536Z namespaces/openshift-cloud-credential-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.442486838Z namespaces/openshift-cloud-credential-operator/monitoring.coreos.com/servicemonitors/cloud-credential-operator.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.442553381Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.442563211Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.442614424Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.442770991Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.442946979Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.443112807Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.443340728Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.443546837Z namespaces/openshift-cloud-network-config-controller/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.44359435Z namespaces/openshift-cloud-network-config-controller/openshift-cloud-network-config-controller.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.443662953Z namespaces/openshift-cloud-network-config-controller/apps.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.443703855Z namespaces/openshift-cloud-network-config-controller/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.443771338Z namespaces/openshift-cloud-network-config-controller/apps/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.443804709Z namespaces/openshift-cloud-network-config-controller/apps/daemonsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.443895054Z namespaces/openshift-cloud-network-config-controller/apps/deployments.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.443966917Z namespaces/openshift-cloud-network-config-controller/apps/replicasets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.444065142Z namespaces/openshift-cloud-network-config-controller/apps/statefulsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.444117304Z namespaces/openshift-cloud-network-config-controller/autoscaling/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.444158236Z namespaces/openshift-cloud-network-config-controller/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.444208279Z namespaces/openshift-cloud-network-config-controller/batch/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.44424081Z namespaces/openshift-cloud-network-config-controller/batch/cronjobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.444325384Z namespaces/openshift-cloud-network-config-controller/batch/jobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.444354795Z namespaces/openshift-cloud-network-config-controller/build.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.444405098Z namespaces/openshift-cloud-network-config-controller/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.444484581Z namespaces/openshift-cloud-network-config-controller/build.openshift.io/builds.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.444531714Z namespaces/openshift-cloud-network-config-controller/coordination.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.444539034Z namespaces/openshift-cloud-network-config-controller/coordination.k8s.io/leases/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.444592927Z namespaces/openshift-cloud-network-config-controller/coordination.k8s.io/leases/cloud-network-config-controller-lock.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.444681541Z namespaces/openshift-cloud-network-config-controller/core/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.444722753Z namespaces/openshift-cloud-network-config-controller/core/configmaps.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.444825498Z namespaces/openshift-cloud-network-config-controller/core/endpoints.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.444905071Z namespaces/openshift-cloud-network-config-controller/core/events.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.445059679Z namespaces/openshift-cloud-network-config-controller/core/persistentvolumeclaims.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.445142363Z namespaces/openshift-cloud-network-config-controller/core/pods.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.445220866Z namespaces/openshift-cloud-network-config-controller/core/replicationcontrollers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.445398485Z namespaces/openshift-cloud-network-config-controller/core/secrets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.445521981Z namespaces/openshift-cloud-network-config-controller/core/services.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.445591744Z namespaces/openshift-cloud-network-config-controller/core/serviceaccounts/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.445643046Z namespaces/openshift-cloud-network-config-controller/core/serviceaccounts/cloud-network-config-controller.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.44572165Z namespaces/openshift-cloud-network-config-controller/discovery.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.445771252Z namespaces/openshift-cloud-network-config-controller/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.445840365Z namespaces/openshift-cloud-network-config-controller/image.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.445887848Z namespaces/openshift-cloud-network-config-controller/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.445985512Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.446080837Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.446398882Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.446505347Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.446553439Z namespaces/openshift-cloud-network-config-controller/monitoring.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.446598911Z namespaces/openshift-cloud-network-config-controller/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.446646223Z namespaces/openshift-cloud-network-config-controller/networking.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.446696596Z namespaces/openshift-cloud-network-config-controller/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.446739118Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.446747248Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.44678368Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.446935487Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.447178318Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.447325415Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.447525624Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.447725194Z namespaces/openshift-cloud-network-config-controller/policy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.447742845Z namespaces/openshift-cloud-network-config-controller/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.447811998Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.44785534Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.447905742Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/rolebindings/cloud-network-config-controller-rb.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.447956165Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.448000157Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/roles/cloud-network-config-controller.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.44807553Z namespaces/openshift-cloud-network-config-controller/route.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.448096102Z namespaces/openshift-cloud-network-config-controller/route.openshift.io/routes.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.448191306Z namespaces/openshift-cluster-csi-drivers/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.448237578Z namespaces/openshift-cluster-csi-drivers/openshift-cluster-csi-drivers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.448290801Z namespaces/openshift-cluster-csi-drivers/apps.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.448337683Z namespaces/openshift-cluster-csi-drivers/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.448421077Z namespaces/openshift-cluster-csi-drivers/apps/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.448444458Z namespaces/openshift-cluster-csi-drivers/apps/daemonsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.448582734Z namespaces/openshift-cluster-csi-drivers/apps/deployments.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.448657028Z namespaces/openshift-cluster-csi-drivers/apps/replicasets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.448737512Z namespaces/openshift-cluster-csi-drivers/apps/statefulsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.448766803Z namespaces/openshift-cluster-csi-drivers/autoscaling/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.448817816Z namespaces/openshift-cluster-csi-drivers/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.448874488Z namespaces/openshift-cluster-csi-drivers/batch/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.44891612Z namespaces/openshift-cluster-csi-drivers/batch/cronjobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.449047466Z namespaces/openshift-cluster-csi-drivers/batch/jobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.449247976Z namespaces/openshift-cluster-csi-drivers/build.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.44934675Z namespaces/openshift-cluster-csi-drivers/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.449448565Z namespaces/openshift-cluster-csi-drivers/build.openshift.io/builds.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.449516038Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.449523729Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.449584401Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/ebs-csi-aws-com.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.449689386Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/external-attacher-leader-ebs-csi-aws-com.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.449795471Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/external-resizer-ebs-csi-aws-com.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.449913957Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/external-snapshotter-leader-ebs-csi-aws-com.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.450246523Z namespaces/openshift-cluster-csi-drivers/core/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.450296885Z namespaces/openshift-cluster-csi-drivers/core/configmaps.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.450422481Z namespaces/openshift-cluster-csi-drivers/core/endpoints.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.450520276Z namespaces/openshift-cluster-csi-drivers/core/events.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.450890743Z namespaces/openshift-cluster-csi-drivers/core/persistentvolumeclaims.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.451029549Z namespaces/openshift-cluster-csi-drivers/core/pods.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.451423128Z namespaces/openshift-cluster-csi-drivers/core/replicationcontrollers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.451580895Z namespaces/openshift-cluster-csi-drivers/core/secrets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.451701351Z namespaces/openshift-cluster-csi-drivers/core/services.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.451733482Z namespaces/openshift-cluster-csi-drivers/core/serviceaccounts/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.451791065Z namespaces/openshift-cluster-csi-drivers/core/serviceaccounts/aws-ebs-csi-driver-operator.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.451842748Z namespaces/openshift-cluster-csi-drivers/discovery.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.451870489Z namespaces/openshift-cluster-csi-drivers/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.451928262Z namespaces/openshift-cluster-csi-drivers/image.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.451974064Z namespaces/openshift-cluster-csi-drivers/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.452020976Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.452080059Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.452153862Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.452235396Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.452284748Z namespaces/openshift-cluster-csi-drivers/monitoring.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.45232481Z namespaces/openshift-cluster-csi-drivers/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.452373083Z namespaces/openshift-cluster-csi-drivers/networking.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.452413025Z namespaces/openshift-cluster-csi-drivers/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.452483188Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.452494108Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.452549491Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.452699358Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.452875966Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.453024453Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.453242514Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.453433362Z namespaces/openshift-cluster-csi-drivers/pods/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.453448083Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-4hjns/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.453494865Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-4hjns/aws-ebs-csi-driver-node-4hjns.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.45360091Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-4hjns/csi-driver/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.453607861Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-4hjns/csi-driver/csi-driver/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.453614501Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-4hjns/csi-driver/csi-driver/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.453659023Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-4hjns/csi-driver/csi-driver/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.453756918Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-4hjns/csi-driver/csi-driver/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.453827751Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-4hjns/csi-driver/csi-driver/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.453869033Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-4hjns/csi-liveness-probe/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.453875974Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-4hjns/csi-liveness-probe/csi-liveness-probe/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.453887714Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-4hjns/csi-liveness-probe/csi-liveness-probe/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.453927476Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-4hjns/csi-liveness-probe/csi-liveness-probe/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.454030191Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-4hjns/csi-liveness-probe/csi-liveness-probe/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.454129695Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-4hjns/csi-liveness-probe/csi-liveness-probe/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.454190888Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-4hjns/csi-node-driver-registrar/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.454200569Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-4hjns/csi-node-driver-registrar/csi-node-driver-registrar/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.454208029Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-4hjns/csi-node-driver-registrar/csi-node-driver-registrar/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.454266002Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-4hjns/csi-node-driver-registrar/csi-node-driver-registrar/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.454399828Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-4hjns/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.454463681Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-4hjns/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.454524674Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-8hl6k/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.454572076Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-8hl6k/aws-ebs-csi-driver-node-8hl6k.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.45465996Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-8hl6k/csi-driver/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.454675651Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-8hl6k/csi-driver/csi-driver/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.454685001Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-8hl6k/csi-driver/csi-driver/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.454756545Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-8hl6k/csi-driver/csi-driver/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.454887621Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-8hl6k/csi-driver/csi-driver/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.454975515Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-8hl6k/csi-driver/csi-driver/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.455127082Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-8hl6k/csi-liveness-probe/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.455138223Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-8hl6k/csi-liveness-probe/csi-liveness-probe/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.455152203Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-8hl6k/csi-liveness-probe/csi-liveness-probe/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.455203236Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-8hl6k/csi-liveness-probe/csi-liveness-probe/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.455357573Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-8hl6k/csi-liveness-probe/csi-liveness-probe/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.45549932Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-8hl6k/csi-liveness-probe/csi-liveness-probe/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.455561643Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-8hl6k/csi-node-driver-registrar/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.455569183Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-8hl6k/csi-node-driver-registrar/csi-node-driver-registrar/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.455575614Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-8hl6k/csi-node-driver-registrar/csi-node-driver-registrar/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.455641886Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-8hl6k/csi-node-driver-registrar/csi-node-driver-registrar/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.455764662Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-8hl6k/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.455849316Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-8hl6k/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.45591479Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-lwg5s/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.455970852Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-lwg5s/aws-ebs-csi-driver-node-lwg5s.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.456099208Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-lwg5s/csi-driver/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.45613543Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-lwg5s/csi-driver/csi-driver/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.456166341Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-lwg5s/csi-driver/csi-driver/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.456293727Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-lwg5s/csi-driver/csi-driver/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.456487056Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-lwg5s/csi-driver/csi-driver/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.456586901Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-lwg5s/csi-driver/csi-driver/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.456647074Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-lwg5s/csi-liveness-probe/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.456656494Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-lwg5s/csi-liveness-probe/csi-liveness-probe/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.456666245Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-lwg5s/csi-liveness-probe/csi-liveness-probe/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.456725937Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-lwg5s/csi-liveness-probe/csi-liveness-probe/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.456835923Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-lwg5s/csi-liveness-probe/csi-liveness-probe/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.456925717Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-lwg5s/csi-liveness-probe/csi-liveness-probe/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.457102015Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-lwg5s/csi-node-driver-registrar/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.457114946Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-lwg5s/csi-node-driver-registrar/csi-node-driver-registrar/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.457119896Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-lwg5s/csi-node-driver-registrar/csi-node-driver-registrar/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.457171478Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-lwg5s/csi-node-driver-registrar/csi-node-driver-registrar/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.457288174Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-lwg5s/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.457385639Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-lwg5s/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.457444051Z namespaces/openshift-cluster-csi-drivers/policy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.457502674Z namespaces/openshift-cluster-csi-drivers/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.457573747Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.457580908Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.45763767Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/rolebindings/aws-ebs-csi-driver-operator-rolebinding.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.457721604Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.457774027Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/roles/aws-ebs-csi-driver-operator-role.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.45784873Z namespaces/openshift-cluster-csi-drivers/route.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.457891552Z namespaces/openshift-cluster-csi-drivers/route.openshift.io/routes.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.457980556Z namespaces/openshift-cluster-machine-approver/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.457987777Z namespaces/openshift-cluster-machine-approver/coordination.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.457991997Z namespaces/openshift-cluster-machine-approver/coordination.k8s.io/leases/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.458067601Z namespaces/openshift-cluster-machine-approver/coordination.k8s.io/leases/cluster-machine-approver-leader.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.458154095Z namespaces/openshift-cluster-machine-approver/monitoring.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.458161555Z namespaces/openshift-cluster-machine-approver/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.458214547Z namespaces/openshift-cluster-machine-approver/monitoring.coreos.com/servicemonitors/cluster-machine-approver.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.458286291Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.458300732Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.458366875Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.458545053Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.458757763Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.458923961Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.459181813Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.459592983Z namespaces/openshift-cluster-node-tuning-operator/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.459658375Z namespaces/openshift-cluster-node-tuning-operator/openshift-cluster-node-tuning-operator.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.459786832Z namespaces/openshift-cluster-node-tuning-operator/apps.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.459833594Z namespaces/openshift-cluster-node-tuning-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.459909167Z namespaces/openshift-cluster-node-tuning-operator/apps/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.459950469Z namespaces/openshift-cluster-node-tuning-operator/apps/daemonsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.460094626Z namespaces/openshift-cluster-node-tuning-operator/apps/deployments.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.46017421Z namespaces/openshift-cluster-node-tuning-operator/apps/replicasets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.460250873Z namespaces/openshift-cluster-node-tuning-operator/apps/statefulsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.460369329Z namespaces/openshift-cluster-node-tuning-operator/apps/daemonsets/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.460416081Z namespaces/openshift-cluster-node-tuning-operator/apps/daemonsets/tuned.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.460500095Z namespaces/openshift-cluster-node-tuning-operator/autoscaling/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.460541587Z namespaces/openshift-cluster-node-tuning-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.460575038Z namespaces/openshift-cluster-node-tuning-operator/batch/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.460624011Z namespaces/openshift-cluster-node-tuning-operator/batch/cronjobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.460708545Z namespaces/openshift-cluster-node-tuning-operator/batch/jobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.460742497Z namespaces/openshift-cluster-node-tuning-operator/build.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.460796079Z namespaces/openshift-cluster-node-tuning-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.460877683Z namespaces/openshift-cluster-node-tuning-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.460921745Z namespaces/openshift-cluster-node-tuning-operator/coordination.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.460934895Z namespaces/openshift-cluster-node-tuning-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.460982128Z namespaces/openshift-cluster-node-tuning-operator/coordination.k8s.io/leases/node-tuning-operator-lock.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.461093363Z namespaces/openshift-cluster-node-tuning-operator/core/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.461124895Z namespaces/openshift-cluster-node-tuning-operator/core/configmaps.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.462029217Z namespaces/openshift-cluster-node-tuning-operator/core/endpoints.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.462173324Z namespaces/openshift-cluster-node-tuning-operator/core/events.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.462344322Z namespaces/openshift-cluster-node-tuning-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.462451517Z namespaces/openshift-cluster-node-tuning-operator/core/pods.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.462682718Z namespaces/openshift-cluster-node-tuning-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.462860676Z namespaces/openshift-cluster-node-tuning-operator/core/secrets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.463115138Z namespaces/openshift-cluster-node-tuning-operator/core/services.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.463202622Z namespaces/openshift-cluster-node-tuning-operator/core/serviceaccounts/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.463256245Z namespaces/openshift-cluster-node-tuning-operator/core/serviceaccounts/cluster-node-tuning-operator.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.463340399Z namespaces/openshift-cluster-node-tuning-operator/discovery.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.46337823Z namespaces/openshift-cluster-node-tuning-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.463469405Z namespaces/openshift-cluster-node-tuning-operator/image.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.463503376Z namespaces/openshift-cluster-node-tuning-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.463599171Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.463653803Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.463769329Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.463867253Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.463945397Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.464030761Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.464148357Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.464191469Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/prometheusrules/node-tuning-operator.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.464289313Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.464349706Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/servicemonitors/node-tuning-operator.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.464415409Z namespaces/openshift-cluster-node-tuning-operator/networking.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.464459091Z namespaces/openshift-cluster-node-tuning-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.464523054Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.464540415Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.464608368Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.464841629Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.465095941Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.465264949Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.465511951Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.465770173Z namespaces/openshift-cluster-node-tuning-operator/pods/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.465782323Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-nhsdc/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.465835986Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-nhsdc/tuned-nhsdc.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.465955612Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-nhsdc/tuned/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.465963862Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-nhsdc/tuned/tuned/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.465975013Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-nhsdc/tuned/tuned/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.466060677Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-nhsdc/tuned/tuned/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.466197273Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-nhsdc/tuned/tuned/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.466294448Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-nhsdc/tuned/tuned/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.466365971Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-rrkxz/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.466427614Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-rrkxz/tuned-rrkxz.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.466524658Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-rrkxz/tuned/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.466532199Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-rrkxz/tuned/tuned/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.466536259Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-rrkxz/tuned/tuned/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.466587012Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-rrkxz/tuned/tuned/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.466715427Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-rrkxz/tuned/tuned/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.466809312Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-rrkxz/tuned/tuned/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.466843153Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-xkgp2/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.466909497Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-xkgp2/tuned-xkgp2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.467019272Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-xkgp2/tuned/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.467032602Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-xkgp2/tuned/tuned/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.467036582Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-xkgp2/tuned/tuned/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.467079555Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-xkgp2/tuned/tuned/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.467219911Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-xkgp2/tuned/tuned/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.467315896Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-xkgp2/tuned/tuned/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.467370438Z namespaces/openshift-cluster-node-tuning-operator/policy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.467420901Z namespaces/openshift-cluster-node-tuning-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.467482274Z namespaces/openshift-cluster-node-tuning-operator/route.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.467536526Z namespaces/openshift-cluster-node-tuning-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.467596649Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.46761261Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.467675683Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ip-10-0-129-80.ec2.internal.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.467794928Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ip-10-0-133-86.ec2.internal.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.467904414Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ip-10-0-140-240.ec2.internal.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.467962966Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/tuneds/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.468021809Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/tuneds/default.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.468177246Z namespaces/openshift-cluster-samples-operator/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.468237759Z namespaces/openshift-cluster-samples-operator/openshift-cluster-samples-operator.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.468323763Z namespaces/openshift-cluster-samples-operator/apps.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.468423988Z namespaces/openshift-cluster-samples-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.468510692Z namespaces/openshift-cluster-samples-operator/apps/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.468560944Z namespaces/openshift-cluster-samples-operator/apps/daemonsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.468663679Z namespaces/openshift-cluster-samples-operator/apps/deployments.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.468804216Z namespaces/openshift-cluster-samples-operator/apps/replicasets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.468926391Z namespaces/openshift-cluster-samples-operator/apps/statefulsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.468997185Z namespaces/openshift-cluster-samples-operator/autoscaling/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.469077919Z namespaces/openshift-cluster-samples-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.469168503Z namespaces/openshift-cluster-samples-operator/batch/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.469225546Z namespaces/openshift-cluster-samples-operator/batch/cronjobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.46932939Z namespaces/openshift-cluster-samples-operator/batch/jobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.469401644Z namespaces/openshift-cluster-samples-operator/build.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.469460427Z namespaces/openshift-cluster-samples-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.469563421Z namespaces/openshift-cluster-samples-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.469658446Z namespaces/openshift-cluster-samples-operator/core/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.469712639Z namespaces/openshift-cluster-samples-operator/core/configmaps.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.469921998Z namespaces/openshift-cluster-samples-operator/core/endpoints.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.470047284Z namespaces/openshift-cluster-samples-operator/core/events.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.470211592Z namespaces/openshift-cluster-samples-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.470307987Z namespaces/openshift-cluster-samples-operator/core/pods.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.470459954Z namespaces/openshift-cluster-samples-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.470619841Z namespaces/openshift-cluster-samples-operator/core/secrets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.470839632Z namespaces/openshift-cluster-samples-operator/core/services.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.470907435Z namespaces/openshift-cluster-samples-operator/discovery.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.470965047Z namespaces/openshift-cluster-samples-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.471533544Z namespaces/openshift-cluster-samples-operator/image.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.471588157Z namespaces/openshift-cluster-samples-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.47166555Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.471710942Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.471794886Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.47187364Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.471920102Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.471963364Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.472016357Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.47208612Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/prometheusrules/samples-operator-alerts.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.472162944Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.472189485Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/servicemonitors/cluster-samples-operator.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.472251558Z namespaces/openshift-cluster-samples-operator/networking.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.472279099Z namespaces/openshift-cluster-samples-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.472337652Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.472348652Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.472375684Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.472531291Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.4727109Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.472847556Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.473077447Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.473270106Z namespaces/openshift-cluster-samples-operator/pods/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.473277326Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-tntpg/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.473315248Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-tntpg/cluster-samples-operator-6748489656-tntpg.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.473409333Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-tntpg/cluster-samples-operator-watch/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.473416473Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-tntpg/cluster-samples-operator-watch/cluster-samples-operator-watch/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.473421093Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-tntpg/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.473471555Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-tntpg/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.47356549Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-tntpg/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.473634103Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-tntpg/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.473689866Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-tntpg/cluster-samples-operator/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.473698636Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-tntpg/cluster-samples-operator/cluster-samples-operator/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.473703786Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-tntpg/cluster-samples-operator/cluster-samples-operator/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.473748689Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-tntpg/cluster-samples-operator/cluster-samples-operator/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.474612559Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-tntpg/cluster-samples-operator/cluster-samples-operator/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.474683013Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-tntpg/cluster-samples-operator/cluster-samples-operator/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.474734125Z namespaces/openshift-cluster-samples-operator/policy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.474765156Z namespaces/openshift-cluster-samples-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.474829759Z namespaces/openshift-cluster-samples-operator/route.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.474886352Z namespaces/openshift-cluster-samples-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.474987377Z namespaces/openshift-cluster-storage-operator/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.4750637Z namespaces/openshift-cluster-storage-operator/openshift-cluster-storage-operator.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.475133544Z namespaces/openshift-cluster-storage-operator/apps.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.475183816Z namespaces/openshift-cluster-storage-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.475248809Z namespaces/openshift-cluster-storage-operator/apps/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.475291951Z namespaces/openshift-cluster-storage-operator/apps/daemonsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.475385446Z namespaces/openshift-cluster-storage-operator/apps/deployments.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.47549256Z namespaces/openshift-cluster-storage-operator/apps/replicasets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.475599716Z namespaces/openshift-cluster-storage-operator/apps/statefulsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.475651148Z namespaces/openshift-cluster-storage-operator/autoscaling/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.47570139Z namespaces/openshift-cluster-storage-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.475756803Z namespaces/openshift-cluster-storage-operator/batch/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.475795345Z namespaces/openshift-cluster-storage-operator/batch/cronjobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.475880309Z namespaces/openshift-cluster-storage-operator/batch/jobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.475929191Z namespaces/openshift-cluster-storage-operator/build.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.475962502Z namespaces/openshift-cluster-storage-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.476076238Z namespaces/openshift-cluster-storage-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.4761253Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.476138581Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.476186193Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/leases/data-source-validator-leader.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.476272717Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/leases/snapshot-controller-leader.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.476357761Z namespaces/openshift-cluster-storage-operator/core/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.476400923Z namespaces/openshift-cluster-storage-operator/core/configmaps.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.476517159Z namespaces/openshift-cluster-storage-operator/core/endpoints.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.476602253Z namespaces/openshift-cluster-storage-operator/core/events.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.476926208Z namespaces/openshift-cluster-storage-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.477018742Z namespaces/openshift-cluster-storage-operator/core/pods.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.477146348Z namespaces/openshift-cluster-storage-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.477295075Z namespaces/openshift-cluster-storage-operator/core/secrets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.477430512Z namespaces/openshift-cluster-storage-operator/core/services.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.477497875Z namespaces/openshift-cluster-storage-operator/core/serviceaccounts/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.477542607Z namespaces/openshift-cluster-storage-operator/core/serviceaccounts/csi-snapshot-controller-operator.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.4776086Z namespaces/openshift-cluster-storage-operator/discovery.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.477666213Z namespaces/openshift-cluster-storage-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.477738126Z namespaces/openshift-cluster-storage-operator/image.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.477782518Z namespaces/openshift-cluster-storage-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.477854842Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.477890893Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.47803284Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.478130124Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.478184467Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.47823725Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.478293912Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.478344615Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/prometheusrules/prometheus.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.478429719Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.478473531Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/servicemonitors/cluster-storage-operator.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.478527323Z namespaces/openshift-cluster-storage-operator/networking.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.478559245Z namespaces/openshift-cluster-storage-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.47865956Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.478687171Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.478737133Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.47888954Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.47910053Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.479242357Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.479449197Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.479643926Z namespaces/openshift-cluster-storage-operator/pods/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.479652866Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-bj476/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.479685468Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-bj476/volume-data-source-validator-549c6d54cf-bj476.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.479768381Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-bj476/volume-data-source-validator/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.479778322Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-bj476/volume-data-source-validator/volume-data-source-validator/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.479783362Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-bj476/volume-data-source-validator/volume-data-source-validator/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.479814874Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-bj476/volume-data-source-validator/volume-data-source-validator/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.479913098Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-bj476/volume-data-source-validator/volume-data-source-validator/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.479998883Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-bj476/volume-data-source-validator/volume-data-source-validator/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.480063766Z namespaces/openshift-cluster-storage-operator/policy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.480100497Z namespaces/openshift-cluster-storage-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.4801561Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.48016481Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.480219823Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/rolebindings/csi-snapshot-controller-operator-role.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.480317758Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.48037836Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/roles/csi-snapshot-controller-operator-role.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.480496276Z namespaces/openshift-cluster-storage-operator/route.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.480558259Z namespaces/openshift-cluster-storage-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.480648103Z namespaces/openshift-cluster-version/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.480683115Z namespaces/openshift-cluster-version/openshift-cluster-version.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.480754878Z namespaces/openshift-cluster-version/apps.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.48079868Z namespaces/openshift-cluster-version/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.480843272Z namespaces/openshift-cluster-version/apps/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.480883884Z namespaces/openshift-cluster-version/apps/daemonsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.480963658Z namespaces/openshift-cluster-version/apps/deployments.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.481076643Z namespaces/openshift-cluster-version/apps/replicasets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.481159907Z namespaces/openshift-cluster-version/apps/statefulsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.48121429Z namespaces/openshift-cluster-version/autoscaling/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.481261622Z namespaces/openshift-cluster-version/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.481320525Z namespaces/openshift-cluster-version/batch/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.481372087Z namespaces/openshift-cluster-version/batch/cronjobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.481470322Z namespaces/openshift-cluster-version/batch/jobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.481530284Z namespaces/openshift-cluster-version/build.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.481559406Z namespaces/openshift-cluster-version/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.48165206Z namespaces/openshift-cluster-version/build.openshift.io/builds.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.481705503Z namespaces/openshift-cluster-version/coordination.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.481715193Z namespaces/openshift-cluster-version/coordination.k8s.io/leases/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.481762015Z namespaces/openshift-cluster-version/coordination.k8s.io/leases/version.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.481839809Z namespaces/openshift-cluster-version/core/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.481895992Z namespaces/openshift-cluster-version/core/configmaps.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.482023668Z namespaces/openshift-cluster-version/core/endpoints.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.482138583Z namespaces/openshift-cluster-version/core/events.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.482259289Z namespaces/openshift-cluster-version/core/persistentvolumeclaims.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.482370194Z namespaces/openshift-cluster-version/core/pods.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.482487349Z namespaces/openshift-cluster-version/core/replicationcontrollers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.482650457Z namespaces/openshift-cluster-version/core/secrets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.482838086Z namespaces/openshift-cluster-version/core/services.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.48291154Z namespaces/openshift-cluster-version/discovery.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.482948911Z namespaces/openshift-cluster-version/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.483019735Z namespaces/openshift-cluster-version/image.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.483084588Z namespaces/openshift-cluster-version/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.4831396Z namespaces/openshift-cluster-version/k8s.ovn.org/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.483166961Z namespaces/openshift-cluster-version/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.483259326Z namespaces/openshift-cluster-version/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.48335004Z namespaces/openshift-cluster-version/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.483383382Z namespaces/openshift-cluster-version/monitoring.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.483433474Z namespaces/openshift-cluster-version/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.483491887Z namespaces/openshift-cluster-version/networking.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.483529279Z namespaces/openshift-cluster-version/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.483584711Z namespaces/openshift-cluster-version/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.483594342Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.483649534Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.483832403Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.484028412Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.484181469Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.484388369Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.484583728Z namespaces/openshift-cluster-version/policy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.48461664Z namespaces/openshift-cluster-version/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.484687953Z namespaces/openshift-cluster-version/route.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.484720905Z namespaces/openshift-cluster-version/route.openshift.io/routes.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.48482671Z namespaces/openshift-config-managed/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.484859471Z namespaces/openshift-config-managed/openshift-config-managed.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.484937225Z namespaces/openshift-config-managed/apps.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.484984467Z namespaces/openshift-config-managed/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.485073281Z namespaces/openshift-config-managed/apps/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.485120343Z namespaces/openshift-config-managed/apps/daemonsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.485217248Z namespaces/openshift-config-managed/apps/deployments.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.485296182Z namespaces/openshift-config-managed/apps/replicasets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.485380056Z namespaces/openshift-config-managed/apps/statefulsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.485418738Z namespaces/openshift-config-managed/autoscaling/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.48546575Z namespaces/openshift-config-managed/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.485506991Z namespaces/openshift-config-managed/batch/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.485574395Z namespaces/openshift-config-managed/batch/cronjobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.48568274Z namespaces/openshift-config-managed/batch/jobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.485745913Z namespaces/openshift-config-managed/build.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.485785345Z namespaces/openshift-config-managed/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.485875339Z namespaces/openshift-config-managed/build.openshift.io/builds.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.485949183Z namespaces/openshift-config-managed/core/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.485994694Z namespaces/openshift-config-managed/core/configmaps.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.489594294Z namespaces/openshift-config-managed/core/endpoints.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.489677248Z namespaces/openshift-config-managed/core/events.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.489800864Z namespaces/openshift-config-managed/core/persistentvolumeclaims.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.489888178Z namespaces/openshift-config-managed/core/pods.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.489966021Z namespaces/openshift-config-managed/core/replicationcontrollers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.49014648Z namespaces/openshift-config-managed/core/secrets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.490256705Z namespaces/openshift-config-managed/core/services.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.490323028Z namespaces/openshift-config-managed/core/configmaps/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.490378631Z namespaces/openshift-config-managed/core/configmaps/console-public.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.490457924Z namespaces/openshift-config-managed/core/configmaps/openshift-network-features.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.490503787Z namespaces/openshift-config-managed/discovery.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.490547669Z namespaces/openshift-config-managed/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.490585381Z namespaces/openshift-config-managed/image.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.490634293Z namespaces/openshift-config-managed/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.490692706Z namespaces/openshift-config-managed/k8s.ovn.org/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.490738398Z namespaces/openshift-config-managed/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.490825972Z namespaces/openshift-config-managed/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.490913886Z namespaces/openshift-config-managed/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.490946427Z namespaces/openshift-config-managed/monitoring.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.49099781Z namespaces/openshift-config-managed/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.491077664Z namespaces/openshift-config-managed/networking.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.491125526Z namespaces/openshift-config-managed/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.491187049Z namespaces/openshift-config-managed/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.491197329Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.491252602Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.491507324Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.491736385Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.491879911Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.492104852Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.492415977Z namespaces/openshift-config-managed/policy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.492466079Z namespaces/openshift-config-managed/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.492520042Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.492527632Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.492577004Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/rolebindings/openshift-network-public-role-binding.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.492648688Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.492676319Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/roles/openshift-network-public-role.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.492746142Z namespaces/openshift-config-managed/route.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.492793154Z namespaces/openshift-config-managed/route.openshift.io/routes.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.492825786Z namespaces/openshift-config-operator/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.492836276Z namespaces/openshift-config-operator/monitoring.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.492845967Z namespaces/openshift-config-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.49290731Z namespaces/openshift-config-operator/monitoring.coreos.com/servicemonitors/config-operator.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.492982063Z namespaces/openshift-config-operator/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.492993804Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.493054167Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.493234725Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.493415974Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.49355384Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.49375711Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.494105896Z namespaces/openshift-config/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.494149908Z namespaces/openshift-config/openshift-config.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.494220492Z namespaces/openshift-config/apps.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.494262674Z namespaces/openshift-config/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.494327506Z namespaces/openshift-config/apps/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.494370039Z namespaces/openshift-config/apps/daemonsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.494458743Z namespaces/openshift-config/apps/deployments.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.494533216Z namespaces/openshift-config/apps/replicasets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.49461175Z namespaces/openshift-config/apps/statefulsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.494671993Z namespaces/openshift-config/autoscaling/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.494702694Z namespaces/openshift-config/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.494778688Z namespaces/openshift-config/batch/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.494806019Z namespaces/openshift-config/batch/cronjobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.494901724Z namespaces/openshift-config/batch/jobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.494947056Z namespaces/openshift-config/build.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.494978877Z namespaces/openshift-config/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.495148855Z namespaces/openshift-config/build.openshift.io/builds.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.495191047Z namespaces/openshift-config/core/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.49523844Z namespaces/openshift-config/core/configmaps.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.495381576Z namespaces/openshift-config/core/endpoints.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.49545267Z namespaces/openshift-config/core/events.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.495531613Z namespaces/openshift-config/core/persistentvolumeclaims.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.495608727Z namespaces/openshift-config/core/pods.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.495691711Z namespaces/openshift-config/core/replicationcontrollers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.495831617Z namespaces/openshift-config/core/secrets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.495970014Z namespaces/openshift-config/core/services.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.496024317Z namespaces/openshift-config/core/secrets/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.496086239Z namespaces/openshift-config/core/secrets/pull-secret.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.496136702Z namespaces/openshift-config/discovery.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.496180184Z namespaces/openshift-config/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.496240317Z namespaces/openshift-config/image.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.496282018Z namespaces/openshift-config/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.496366173Z namespaces/openshift-config/k8s.ovn.org/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.496405895Z namespaces/openshift-config/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.496492319Z namespaces/openshift-config/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.496569672Z namespaces/openshift-config/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.496616254Z namespaces/openshift-config/monitoring.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.496654606Z namespaces/openshift-config/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.496709109Z namespaces/openshift-config/networking.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.496757571Z namespaces/openshift-config/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.496814804Z namespaces/openshift-config/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.496824514Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.496873337Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.497052375Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.497225383Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.49737881Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.49757421Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.497763148Z namespaces/openshift-config/policy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.49779321Z namespaces/openshift-config/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.497850452Z namespaces/openshift-config/route.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.497890464Z namespaces/openshift-config/route.openshift.io/routes.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.497983479Z namespaces/openshift-console-operator/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.498050662Z namespaces/openshift-console-operator/openshift-console-operator.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.498166617Z namespaces/openshift-console-operator/apps.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.498210929Z namespaces/openshift-console-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.498285383Z namespaces/openshift-console-operator/apps/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.498322685Z namespaces/openshift-console-operator/apps/daemonsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.498412149Z namespaces/openshift-console-operator/apps/deployments.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.498526954Z namespaces/openshift-console-operator/apps/replicasets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.498630539Z namespaces/openshift-console-operator/apps/statefulsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.498676231Z namespaces/openshift-console-operator/autoscaling/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.498715713Z namespaces/openshift-console-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.498768286Z namespaces/openshift-console-operator/batch/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.498813548Z namespaces/openshift-console-operator/batch/cronjobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.498895512Z namespaces/openshift-console-operator/batch/jobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.498938854Z namespaces/openshift-console-operator/build.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.498984486Z namespaces/openshift-console-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.49908651Z namespaces/openshift-console-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.499187885Z namespaces/openshift-console-operator/coordination.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.499194935Z namespaces/openshift-console-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.499225877Z namespaces/openshift-console-operator/coordination.k8s.io/leases/console-operator-lock.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.499317351Z namespaces/openshift-console-operator/core/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.499349853Z namespaces/openshift-console-operator/core/configmaps.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.499952731Z namespaces/openshift-console-operator/core/endpoints.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.500071887Z namespaces/openshift-console-operator/core/events.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.50034226Z namespaces/openshift-console-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.500440404Z namespaces/openshift-console-operator/core/pods.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.500606812Z namespaces/openshift-console-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.50077409Z namespaces/openshift-console-operator/core/secrets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.50099588Z namespaces/openshift-console-operator/core/services.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.501079734Z namespaces/openshift-console-operator/discovery.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.501123996Z namespaces/openshift-console-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.501184659Z namespaces/openshift-console-operator/image.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.501219941Z namespaces/openshift-console-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.501289594Z namespaces/openshift-console-operator/k8s.ovn.org/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.501342747Z namespaces/openshift-console-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.501454052Z namespaces/openshift-console-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.501534256Z namespaces/openshift-console-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.501601209Z namespaces/openshift-console-operator/monitoring.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.501643561Z namespaces/openshift-console-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.501696823Z namespaces/openshift-console-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.501738155Z namespaces/openshift-console-operator/monitoring.coreos.com/prometheusrules/cluster-monitoring-prometheus-rules.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.501805129Z namespaces/openshift-console-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.50184488Z namespaces/openshift-console-operator/monitoring.coreos.com/servicemonitors/console-operator.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.501897743Z namespaces/openshift-console-operator/networking.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.501937255Z namespaces/openshift-console-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.501971736Z namespaces/openshift-console-operator/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.501997558Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.502089682Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.502380696Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.502558294Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.50269708Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.502894Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.503241766Z namespaces/openshift-console-operator/pods/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.503252826Z namespaces/openshift-console-operator/pods/console-operator-584dcb6b54-bmxdx/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.503297599Z namespaces/openshift-console-operator/pods/console-operator-584dcb6b54-bmxdx/console-operator-584dcb6b54-bmxdx.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.503408044Z namespaces/openshift-console-operator/pods/console-operator-584dcb6b54-bmxdx/console-operator/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.503417234Z namespaces/openshift-console-operator/pods/console-operator-584dcb6b54-bmxdx/console-operator/console-operator/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.503421885Z namespaces/openshift-console-operator/pods/console-operator-584dcb6b54-bmxdx/console-operator/console-operator/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.503462607Z namespaces/openshift-console-operator/pods/console-operator-584dcb6b54-bmxdx/console-operator/console-operator/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.50397133Z namespaces/openshift-console-operator/pods/console-operator-584dcb6b54-bmxdx/console-operator/console-operator/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.504019582Z namespaces/openshift-console-operator/policy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.504100316Z namespaces/openshift-console-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.50418219Z namespaces/openshift-console-operator/route.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.504247993Z namespaces/openshift-console-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.504327467Z namespaces/openshift-console-user-settings/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.504335848Z namespaces/openshift-console-user-settings/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.504343918Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.504417311Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.504655292Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.504839741Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.504982368Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.505208009Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.505415168Z namespaces/openshift-console/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.50546317Z namespaces/openshift-console/openshift-console.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.50566985Z namespaces/openshift-console/apps.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.505709182Z namespaces/openshift-console/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.505775325Z namespaces/openshift-console/apps/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.505822638Z namespaces/openshift-console/apps/daemonsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.505900481Z namespaces/openshift-console/apps/deployments.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.50609247Z namespaces/openshift-console/apps/replicasets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.506353103Z namespaces/openshift-console/apps/statefulsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.506393114Z namespaces/openshift-console/autoscaling/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.506445847Z namespaces/openshift-console/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.50650969Z namespaces/openshift-console/batch/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.506556632Z namespaces/openshift-console/batch/cronjobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.506664767Z namespaces/openshift-console/batch/jobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.50673354Z namespaces/openshift-console/build.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.506781032Z namespaces/openshift-console/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.506861786Z namespaces/openshift-console/build.openshift.io/builds.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.506925359Z namespaces/openshift-console/core/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.506982952Z namespaces/openshift-console/core/configmaps.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.507642413Z namespaces/openshift-console/core/endpoints.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.507728607Z namespaces/openshift-console/core/events.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.507980389Z namespaces/openshift-console/core/persistentvolumeclaims.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.508082984Z namespaces/openshift-console/core/pods.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.508284243Z namespaces/openshift-console/core/replicationcontrollers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.508411319Z namespaces/openshift-console/core/secrets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.508596588Z namespaces/openshift-console/core/services.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.508656901Z namespaces/openshift-console/discovery.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.508683932Z namespaces/openshift-console/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.508762686Z namespaces/openshift-console/image.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.508794117Z namespaces/openshift-console/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.508871101Z namespaces/openshift-console/k8s.ovn.org/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.508902782Z namespaces/openshift-console/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.508990136Z namespaces/openshift-console/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.509109682Z namespaces/openshift-console/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.509174105Z namespaces/openshift-console/monitoring.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.509212737Z namespaces/openshift-console/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.50927322Z namespaces/openshift-console/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.509323652Z namespaces/openshift-console/monitoring.coreos.com/servicemonitors/console.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.509381515Z namespaces/openshift-console/networking.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.509415776Z namespaces/openshift-console/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.50948979Z namespaces/openshift-console/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.509504701Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.509570814Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.509801785Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.510001724Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.510161842Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.510359001Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.510569741Z namespaces/openshift-console/pods/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.510583751Z namespaces/openshift-console/pods/console-b9cc94695-plmjd/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.510636644Z namespaces/openshift-console/pods/console-b9cc94695-plmjd/console-b9cc94695-plmjd.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.510731278Z namespaces/openshift-console/pods/console-b9cc94695-plmjd/console/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.510739409Z namespaces/openshift-console/pods/console-b9cc94695-plmjd/console/console/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.510743449Z namespaces/openshift-console/pods/console-b9cc94695-plmjd/console/console/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.510778931Z namespaces/openshift-console/pods/console-b9cc94695-plmjd/console/console/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.510880645Z namespaces/openshift-console/pods/console-b9cc94695-plmjd/console/console/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.5109753Z namespaces/openshift-console/pods/console-b9cc94695-plmjd/console/console/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.511118307Z namespaces/openshift-console/pods/downloads-7c97dc4555-6kv2b/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.511163849Z namespaces/openshift-console/pods/downloads-7c97dc4555-6kv2b/downloads-7c97dc4555-6kv2b.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.511259773Z namespaces/openshift-console/pods/downloads-7c97dc4555-6kv2b/download-server/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.511267514Z namespaces/openshift-console/pods/downloads-7c97dc4555-6kv2b/download-server/download-server/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.511271664Z namespaces/openshift-console/pods/downloads-7c97dc4555-6kv2b/download-server/download-server/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.511314966Z namespaces/openshift-console/pods/downloads-7c97dc4555-6kv2b/download-server/download-server/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.511734696Z namespaces/openshift-console/pods/downloads-7c97dc4555-6kv2b/download-server/download-server/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.51182346Z namespaces/openshift-console/pods/downloads-7c97dc4555-6kv2b/download-server/download-server/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.511883553Z namespaces/openshift-console/policy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.511931085Z namespaces/openshift-console/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.511996948Z namespaces/openshift-console/route.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.512098083Z namespaces/openshift-console/route.openshift.io/routes.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.512221639Z namespaces/openshift-controller-manager-operator/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.512278221Z namespaces/openshift-controller-manager-operator/openshift-controller-manager-operator.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.512353125Z namespaces/openshift-controller-manager-operator/apps.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.512407447Z namespaces/openshift-controller-manager-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.512488521Z namespaces/openshift-controller-manager-operator/apps/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.512509122Z namespaces/openshift-controller-manager-operator/apps/daemonsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.512626007Z namespaces/openshift-controller-manager-operator/apps/deployments.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.512794435Z namespaces/openshift-controller-manager-operator/apps/replicasets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.512878239Z namespaces/openshift-controller-manager-operator/apps/statefulsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.513076969Z namespaces/openshift-controller-manager-operator/autoscaling/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.51311677Z namespaces/openshift-controller-manager-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.513187674Z namespaces/openshift-controller-manager-operator/batch/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.513258387Z namespaces/openshift-controller-manager-operator/batch/cronjobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.513420445Z namespaces/openshift-controller-manager-operator/batch/jobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.513477078Z namespaces/openshift-controller-manager-operator/build.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.513507529Z namespaces/openshift-controller-manager-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.513605734Z namespaces/openshift-controller-manager-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.513674767Z namespaces/openshift-controller-manager-operator/core/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.513719759Z namespaces/openshift-controller-manager-operator/core/configmaps.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.513841215Z namespaces/openshift-controller-manager-operator/core/endpoints.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.513917298Z namespaces/openshift-controller-manager-operator/core/events.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.514028153Z namespaces/openshift-controller-manager-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.514107687Z namespaces/openshift-controller-manager-operator/core/pods.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.514186431Z namespaces/openshift-controller-manager-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.514307507Z namespaces/openshift-controller-manager-operator/core/secrets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.514517236Z namespaces/openshift-controller-manager-operator/core/services.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.51458796Z namespaces/openshift-controller-manager-operator/discovery.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.514654913Z namespaces/openshift-controller-manager-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.514734027Z namespaces/openshift-controller-manager-operator/image.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.51479833Z namespaces/openshift-controller-manager-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.514876613Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.514909645Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.514997629Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.515112715Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.515152126Z namespaces/openshift-controller-manager-operator/monitoring.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.515195948Z namespaces/openshift-controller-manager-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.51523065Z namespaces/openshift-controller-manager-operator/networking.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.515297903Z namespaces/openshift-controller-manager-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.515345515Z namespaces/openshift-controller-manager-operator/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.515356656Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.515405268Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.515553115Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.515982476Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.516145593Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.516341102Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.516526951Z namespaces/openshift-controller-manager-operator/policy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.516582624Z namespaces/openshift-controller-manager-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.516661488Z namespaces/openshift-controller-manager-operator/route.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.5167146Z namespaces/openshift-controller-manager-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.516845246Z namespaces/openshift-controller-manager/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.516906719Z namespaces/openshift-controller-manager/openshift-controller-manager.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.516992053Z namespaces/openshift-controller-manager/apps.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.517050276Z namespaces/openshift-controller-manager/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.517113559Z namespaces/openshift-controller-manager/apps/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.517154401Z namespaces/openshift-controller-manager/apps/daemonsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.517235275Z namespaces/openshift-controller-manager/apps/deployments.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.517309978Z namespaces/openshift-controller-manager/apps/replicasets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.517393592Z namespaces/openshift-controller-manager/apps/statefulsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.517476056Z namespaces/openshift-controller-manager/autoscaling/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.517522238Z namespaces/openshift-controller-manager/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.517587981Z namespaces/openshift-controller-manager/batch/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.517627793Z namespaces/openshift-controller-manager/batch/cronjobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.517711867Z namespaces/openshift-controller-manager/batch/jobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.517757559Z namespaces/openshift-controller-manager/build.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.517798031Z namespaces/openshift-controller-manager/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.517875654Z namespaces/openshift-controller-manager/build.openshift.io/builds.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.517932697Z namespaces/openshift-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.517970089Z namespaces/openshift-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.518103895Z namespaces/openshift-controller-manager/coordination.k8s.io/leases/openshift-master-controllers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.518185889Z namespaces/openshift-controller-manager/core/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.518229981Z namespaces/openshift-controller-manager/core/configmaps.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.518339646Z namespaces/openshift-controller-manager/core/endpoints.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.51841586Z namespaces/openshift-controller-manager/core/events.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.518498664Z namespaces/openshift-controller-manager/core/persistentvolumeclaims.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.518578867Z namespaces/openshift-controller-manager/core/pods.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.518666232Z namespaces/openshift-controller-manager/core/replicationcontrollers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.518790617Z namespaces/openshift-controller-manager/core/secrets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.518981186Z namespaces/openshift-controller-manager/core/services.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.519406897Z namespaces/openshift-controller-manager/discovery.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.519468169Z namespaces/openshift-controller-manager/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.519543373Z namespaces/openshift-controller-manager/image.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.519589265Z namespaces/openshift-controller-manager/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.519644768Z namespaces/openshift-controller-manager/k8s.ovn.org/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.519677479Z namespaces/openshift-controller-manager/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.519761783Z namespaces/openshift-controller-manager/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.519836017Z namespaces/openshift-controller-manager/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.519882969Z namespaces/openshift-controller-manager/monitoring.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.519924081Z namespaces/openshift-controller-manager/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.519972403Z namespaces/openshift-controller-manager/networking.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.520022866Z namespaces/openshift-controller-manager/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.520080428Z namespaces/openshift-controller-manager/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.520088969Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.52012502Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.520272707Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.520456216Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.520591282Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.520779781Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.52096443Z namespaces/openshift-controller-manager/policy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.521022123Z namespaces/openshift-controller-manager/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.521093446Z namespaces/openshift-controller-manager/route.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.521135138Z namespaces/openshift-controller-manager/route.openshift.io/routes.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.521220222Z namespaces/openshift-dns-operator/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.521282595Z namespaces/openshift-dns-operator/openshift-dns-operator.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.521351648Z namespaces/openshift-dns-operator/apps.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.52139252Z namespaces/openshift-dns-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.521458103Z namespaces/openshift-dns-operator/apps/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.521510686Z namespaces/openshift-dns-operator/apps/daemonsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.521627431Z namespaces/openshift-dns-operator/apps/deployments.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.521718975Z namespaces/openshift-dns-operator/apps/replicasets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.521800009Z namespaces/openshift-dns-operator/apps/statefulsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.521856862Z namespaces/openshift-dns-operator/autoscaling/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.521904074Z namespaces/openshift-dns-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.521949676Z namespaces/openshift-dns-operator/batch/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.521990978Z namespaces/openshift-dns-operator/batch/cronjobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.522093013Z namespaces/openshift-dns-operator/batch/jobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.522148136Z namespaces/openshift-dns-operator/build.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.522185197Z namespaces/openshift-dns-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.522269701Z namespaces/openshift-dns-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.522344765Z namespaces/openshift-dns-operator/core/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.522389427Z namespaces/openshift-dns-operator/core/configmaps.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.522490042Z namespaces/openshift-dns-operator/core/endpoints.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.522569545Z namespaces/openshift-dns-operator/core/events.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.52265739Z namespaces/openshift-dns-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.522736823Z namespaces/openshift-dns-operator/core/pods.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.522815037Z namespaces/openshift-dns-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.522934632Z namespaces/openshift-dns-operator/core/secrets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.523134532Z namespaces/openshift-dns-operator/core/services.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.523211276Z namespaces/openshift-dns-operator/discovery.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.523255418Z namespaces/openshift-dns-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.523328421Z namespaces/openshift-dns-operator/image.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.523357092Z namespaces/openshift-dns-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.523417395Z namespaces/openshift-dns-operator/k8s.ovn.org/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.523442356Z namespaces/openshift-dns-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.523533231Z namespaces/openshift-dns-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.523609504Z namespaces/openshift-dns-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.523658617Z namespaces/openshift-dns-operator/monitoring.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.523692178Z namespaces/openshift-dns-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.523749011Z namespaces/openshift-dns-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.523779972Z namespaces/openshift-dns-operator/monitoring.coreos.com/prometheusrules/dns.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.523841305Z namespaces/openshift-dns-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.523872097Z namespaces/openshift-dns-operator/monitoring.coreos.com/servicemonitors/dns-operator.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.523932239Z namespaces/openshift-dns-operator/networking.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.523956601Z namespaces/openshift-dns-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.524014633Z namespaces/openshift-dns-operator/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.524025004Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.524089687Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.524282766Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.524492796Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.524633723Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.524835232Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.525055262Z namespaces/openshift-dns-operator/policy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.525102054Z namespaces/openshift-dns-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.525162537Z namespaces/openshift-dns-operator/route.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.525201289Z namespaces/openshift-dns-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.525318995Z namespaces/openshift-dns/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.525365707Z namespaces/openshift-dns/openshift-dns.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.52542535Z namespaces/openshift-dns/apps.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.525453261Z namespaces/openshift-dns/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.526304661Z namespaces/openshift-dns/apps/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.526363384Z namespaces/openshift-dns/apps/daemonsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.526550893Z namespaces/openshift-dns/apps/deployments.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.526627486Z namespaces/openshift-dns/apps/replicasets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.52670795Z namespaces/openshift-dns/apps/statefulsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.526747852Z namespaces/openshift-dns/autoscaling/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.526792864Z namespaces/openshift-dns/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.526849967Z namespaces/openshift-dns/batch/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.526893549Z namespaces/openshift-dns/batch/cronjobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.526981713Z namespaces/openshift-dns/batch/jobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.52714323Z namespaces/openshift-dns/build.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.527186992Z namespaces/openshift-dns/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.527273377Z namespaces/openshift-dns/build.openshift.io/builds.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.52735239Z namespaces/openshift-dns/core/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.527396633Z namespaces/openshift-dns/core/configmaps.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.527504567Z namespaces/openshift-dns/core/endpoints.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.527590111Z namespaces/openshift-dns/core/events.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.527815252Z namespaces/openshift-dns/core/persistentvolumeclaims.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.527887736Z namespaces/openshift-dns/core/pods.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.528183179Z namespaces/openshift-dns/core/replicationcontrollers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.528308355Z namespaces/openshift-dns/core/secrets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.528475293Z namespaces/openshift-dns/core/services.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.528541796Z namespaces/openshift-dns/discovery.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.528579498Z namespaces/openshift-dns/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.528656792Z namespaces/openshift-dns/image.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.528700344Z namespaces/openshift-dns/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.528769027Z namespaces/openshift-dns/k8s.ovn.org/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.528800728Z namespaces/openshift-dns/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.528899623Z namespaces/openshift-dns/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.528987247Z namespaces/openshift-dns/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.529075612Z namespaces/openshift-dns/monitoring.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.529116734Z namespaces/openshift-dns/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.529178206Z namespaces/openshift-dns/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.529218988Z namespaces/openshift-dns/monitoring.coreos.com/servicemonitors/dns-default.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.529269381Z namespaces/openshift-dns/networking.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.529324603Z namespaces/openshift-dns/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.529386326Z namespaces/openshift-dns/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.529395336Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.529441959Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.529622597Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.529942972Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.53010405Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.530302759Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.530515179Z namespaces/openshift-dns/pods/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.53052958Z namespaces/openshift-dns/pods/dns-default-j5t5h/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.530576072Z namespaces/openshift-dns/pods/dns-default-j5t5h/dns-default-j5t5h.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.530679307Z namespaces/openshift-dns/pods/dns-default-j5t5h/dns/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.530691588Z namespaces/openshift-dns/pods/dns-default-j5t5h/dns/dns/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.530696838Z namespaces/openshift-dns/pods/dns-default-j5t5h/dns/dns/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.53073951Z namespaces/openshift-dns/pods/dns-default-j5t5h/dns/dns/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.530835114Z namespaces/openshift-dns/pods/dns-default-j5t5h/dns/dns/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.530902118Z namespaces/openshift-dns/pods/dns-default-j5t5h/dns/dns/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.530922459Z namespaces/openshift-dns/pods/dns-default-j5t5h/kube-rbac-proxy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.53095711Z namespaces/openshift-dns/pods/dns-default-j5t5h/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.53096595Z namespaces/openshift-dns/pods/dns-default-j5t5h/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.531024243Z namespaces/openshift-dns/pods/dns-default-j5t5h/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.531123508Z namespaces/openshift-dns/pods/dns-default-j5t5h/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.531192281Z namespaces/openshift-dns/pods/dns-default-j5t5h/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.531218992Z namespaces/openshift-dns/pods/dns-default-vgmp2/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.531267345Z namespaces/openshift-dns/pods/dns-default-vgmp2/dns-default-vgmp2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.531361679Z namespaces/openshift-dns/pods/dns-default-vgmp2/dns/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.53137034Z namespaces/openshift-dns/pods/dns-default-vgmp2/dns/dns/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.53138005Z namespaces/openshift-dns/pods/dns-default-vgmp2/dns/dns/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.531434032Z namespaces/openshift-dns/pods/dns-default-vgmp2/dns/dns/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.531526557Z namespaces/openshift-dns/pods/dns-default-vgmp2/dns/dns/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.53159718Z namespaces/openshift-dns/pods/dns-default-vgmp2/dns/dns/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.531642782Z namespaces/openshift-dns/pods/dns-default-vgmp2/kube-rbac-proxy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.531650803Z namespaces/openshift-dns/pods/dns-default-vgmp2/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.531657333Z namespaces/openshift-dns/pods/dns-default-vgmp2/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.531765598Z namespaces/openshift-dns/pods/dns-default-vgmp2/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.531856582Z namespaces/openshift-dns/pods/dns-default-vgmp2/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.531924795Z namespaces/openshift-dns/pods/dns-default-vgmp2/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.531985268Z namespaces/openshift-dns/pods/dns-default-z2cth/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.532054692Z namespaces/openshift-dns/pods/dns-default-z2cth/dns-default-z2cth.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.532130865Z namespaces/openshift-dns/pods/dns-default-z2cth/dns/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.532138496Z namespaces/openshift-dns/pods/dns-default-z2cth/dns/dns/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.532145296Z namespaces/openshift-dns/pods/dns-default-z2cth/dns/dns/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.532198958Z namespaces/openshift-dns/pods/dns-default-z2cth/dns/dns/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.532286572Z namespaces/openshift-dns/pods/dns-default-z2cth/dns/dns/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.532354326Z namespaces/openshift-dns/pods/dns-default-z2cth/dns/dns/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.532395858Z namespaces/openshift-dns/pods/dns-default-z2cth/kube-rbac-proxy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.532404088Z namespaces/openshift-dns/pods/dns-default-z2cth/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.532411248Z namespaces/openshift-dns/pods/dns-default-z2cth/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.532465961Z namespaces/openshift-dns/pods/dns-default-z2cth/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.532557455Z namespaces/openshift-dns/pods/dns-default-z2cth/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.532625039Z namespaces/openshift-dns/pods/dns-default-z2cth/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.53266653Z namespaces/openshift-dns/pods/node-resolver-57dwz/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.532695012Z namespaces/openshift-dns/pods/node-resolver-57dwz/node-resolver-57dwz.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.532798087Z namespaces/openshift-dns/pods/node-resolver-57dwz/dns-node-resolver/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.532806037Z namespaces/openshift-dns/pods/node-resolver-57dwz/dns-node-resolver/dns-node-resolver/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.532814847Z namespaces/openshift-dns/pods/node-resolver-57dwz/dns-node-resolver/dns-node-resolver/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.532857559Z namespaces/openshift-dns/pods/node-resolver-57dwz/dns-node-resolver/dns-node-resolver/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.532960564Z namespaces/openshift-dns/pods/node-resolver-57dwz/dns-node-resolver/dns-node-resolver/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.53307636Z namespaces/openshift-dns/pods/node-resolver-57dwz/dns-node-resolver/dns-node-resolver/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.533150273Z namespaces/openshift-dns/pods/node-resolver-9qhbp/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.533212806Z namespaces/openshift-dns/pods/node-resolver-9qhbp/node-resolver-9qhbp.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.533333602Z namespaces/openshift-dns/pods/node-resolver-9qhbp/dns-node-resolver/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.533341152Z namespaces/openshift-dns/pods/node-resolver-9qhbp/dns-node-resolver/dns-node-resolver/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.533345312Z namespaces/openshift-dns/pods/node-resolver-9qhbp/dns-node-resolver/dns-node-resolver/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.533397275Z namespaces/openshift-dns/pods/node-resolver-9qhbp/dns-node-resolver/dns-node-resolver/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.533491209Z namespaces/openshift-dns/pods/node-resolver-9qhbp/dns-node-resolver/dns-node-resolver/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.533559323Z namespaces/openshift-dns/pods/node-resolver-9qhbp/dns-node-resolver/dns-node-resolver/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.533609275Z namespaces/openshift-dns/pods/node-resolver-gl58w/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.533665348Z namespaces/openshift-dns/pods/node-resolver-gl58w/node-resolver-gl58w.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.533753552Z namespaces/openshift-dns/pods/node-resolver-gl58w/dns-node-resolver/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.533760992Z namespaces/openshift-dns/pods/node-resolver-gl58w/dns-node-resolver/dns-node-resolver/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.533765072Z namespaces/openshift-dns/pods/node-resolver-gl58w/dns-node-resolver/dns-node-resolver/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.533804944Z namespaces/openshift-dns/pods/node-resolver-gl58w/dns-node-resolver/dns-node-resolver/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.533894098Z namespaces/openshift-dns/pods/node-resolver-gl58w/dns-node-resolver/dns-node-resolver/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.533962371Z namespaces/openshift-dns/pods/node-resolver-gl58w/dns-node-resolver/dns-node-resolver/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.533989643Z namespaces/openshift-dns/policy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.534064796Z namespaces/openshift-dns/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.534103198Z namespaces/openshift-dns/route.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.53414913Z namespaces/openshift-dns/route.openshift.io/routes.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.534252685Z namespaces/openshift-etcd/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.534295777Z namespaces/openshift-etcd/openshift-etcd.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.53435964Z namespaces/openshift-etcd/apps.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.534385811Z namespaces/openshift-etcd/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.534458115Z namespaces/openshift-etcd/apps/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.534519028Z namespaces/openshift-etcd/apps/daemonsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.534622633Z namespaces/openshift-etcd/apps/deployments.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.534711367Z namespaces/openshift-etcd/apps/replicasets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.534832652Z namespaces/openshift-etcd/apps/statefulsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.534913436Z namespaces/openshift-etcd/autoscaling/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.534967439Z namespaces/openshift-etcd/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.535016831Z namespaces/openshift-etcd/batch/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.535180069Z namespaces/openshift-etcd/batch/cronjobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.535260963Z namespaces/openshift-etcd/batch/jobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.535331606Z namespaces/openshift-etcd/build.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.535373288Z namespaces/openshift-etcd/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.535455952Z namespaces/openshift-etcd/build.openshift.io/builds.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.535526345Z namespaces/openshift-etcd/core/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.535574937Z namespaces/openshift-etcd/core/configmaps.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.535668952Z namespaces/openshift-etcd/core/endpoints.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.535751866Z namespaces/openshift-etcd/core/events.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.535866421Z namespaces/openshift-etcd/core/persistentvolumeclaims.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.535962096Z namespaces/openshift-etcd/core/pods.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.536066981Z namespaces/openshift-etcd/core/replicationcontrollers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.536187146Z namespaces/openshift-etcd/core/secrets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.536346234Z namespaces/openshift-etcd/core/services.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.536411097Z namespaces/openshift-etcd/discovery.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.536457159Z namespaces/openshift-etcd/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.536504361Z namespaces/openshift-etcd/image.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.536546383Z namespaces/openshift-etcd/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.536595385Z namespaces/openshift-etcd/k8s.ovn.org/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.536657348Z namespaces/openshift-etcd/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.536751503Z namespaces/openshift-etcd/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.536830756Z namespaces/openshift-etcd/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.536893589Z namespaces/openshift-etcd/monitoring.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.536941842Z namespaces/openshift-etcd/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.536988584Z namespaces/openshift-etcd/networking.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.537062557Z namespaces/openshift-etcd/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.537105309Z namespaces/openshift-etcd/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.53711601Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.537171072Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.537320269Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.537501968Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.537646995Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.537916067Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.538133838Z namespaces/openshift-etcd/policy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.538169499Z namespaces/openshift-etcd/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.538234823Z namespaces/openshift-etcd/route.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.538347088Z namespaces/openshift-etcd/route.openshift.io/routes.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.538489384Z namespaces/openshift-host-network/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.538555518Z namespaces/openshift-host-network/openshift-host-network.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.538632201Z namespaces/openshift-host-network/apps.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.538661212Z namespaces/openshift-host-network/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.538738826Z namespaces/openshift-host-network/apps/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.538790658Z namespaces/openshift-host-network/apps/daemonsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.538894094Z namespaces/openshift-host-network/apps/deployments.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.538981218Z namespaces/openshift-host-network/apps/replicasets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.539086203Z namespaces/openshift-host-network/apps/statefulsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.539138455Z namespaces/openshift-host-network/autoscaling/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.539190567Z namespaces/openshift-host-network/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.53924529Z namespaces/openshift-host-network/batch/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.539305163Z namespaces/openshift-host-network/batch/cronjobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.539383477Z namespaces/openshift-host-network/batch/jobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.539443129Z namespaces/openshift-host-network/build.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.539475741Z namespaces/openshift-host-network/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.539563815Z namespaces/openshift-host-network/build.openshift.io/builds.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.539639529Z namespaces/openshift-host-network/core/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.539685401Z namespaces/openshift-host-network/core/configmaps.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.539783585Z namespaces/openshift-host-network/core/endpoints.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.539860729Z namespaces/openshift-host-network/core/events.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.539941903Z namespaces/openshift-host-network/core/persistentvolumeclaims.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.540018896Z namespaces/openshift-host-network/core/pods.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.540125231Z namespaces/openshift-host-network/core/replicationcontrollers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.540275199Z namespaces/openshift-host-network/core/secrets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.540415605Z namespaces/openshift-host-network/core/services.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.540477658Z namespaces/openshift-host-network/core/resourcequotas/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.54052449Z namespaces/openshift-host-network/core/resourcequotas/host-network-namespace-quotas.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.540589633Z namespaces/openshift-host-network/discovery.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.540638716Z namespaces/openshift-host-network/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.540696068Z namespaces/openshift-host-network/image.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.540743781Z namespaces/openshift-host-network/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.540805923Z namespaces/openshift-host-network/k8s.ovn.org/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.540847275Z namespaces/openshift-host-network/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.540932109Z namespaces/openshift-host-network/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.541026104Z namespaces/openshift-host-network/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.541082056Z namespaces/openshift-host-network/monitoring.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.541130708Z namespaces/openshift-host-network/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.541178561Z namespaces/openshift-host-network/networking.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.541220543Z namespaces/openshift-host-network/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.541267015Z namespaces/openshift-host-network/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.541276946Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.541330128Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.541485305Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.541673024Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.5418084Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.5420176Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.54222558Z namespaces/openshift-host-network/policy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.542269712Z namespaces/openshift-host-network/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.542318495Z namespaces/openshift-host-network/route.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.542361477Z namespaces/openshift-host-network/route.openshift.io/routes.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.54243884Z namespaces/openshift-image-registry/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.542486963Z namespaces/openshift-image-registry/openshift-image-registry.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.542549135Z namespaces/openshift-image-registry/apps.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.542581927Z namespaces/openshift-image-registry/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.54263693Z namespaces/openshift-image-registry/apps/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.542664821Z namespaces/openshift-image-registry/apps/daemonsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.542779296Z namespaces/openshift-image-registry/apps/deployments.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.542913503Z namespaces/openshift-image-registry/apps/replicasets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.543337383Z namespaces/openshift-image-registry/apps/statefulsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.543383645Z namespaces/openshift-image-registry/autoscaling/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.543428707Z namespaces/openshift-image-registry/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.543474799Z namespaces/openshift-image-registry/batch/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.543514701Z namespaces/openshift-image-registry/batch/cronjobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.543646547Z namespaces/openshift-image-registry/batch/jobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.54371655Z namespaces/openshift-image-registry/build.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.543783504Z namespaces/openshift-image-registry/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.543895539Z namespaces/openshift-image-registry/build.openshift.io/builds.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.543948371Z namespaces/openshift-image-registry/coordination.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.543957702Z namespaces/openshift-image-registry/coordination.k8s.io/leases/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.543995754Z namespaces/openshift-image-registry/coordination.k8s.io/leases/openshift-master-controllers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.544104639Z namespaces/openshift-image-registry/core/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.54413516Z namespaces/openshift-image-registry/core/configmaps.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.54520036Z namespaces/openshift-image-registry/core/endpoints.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.545308365Z namespaces/openshift-image-registry/core/events.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.545591898Z namespaces/openshift-image-registry/core/persistentvolumeclaims.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.545672602Z namespaces/openshift-image-registry/core/pods.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.545906333Z namespaces/openshift-image-registry/core/replicationcontrollers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.546074011Z namespaces/openshift-image-registry/core/secrets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.546274091Z namespaces/openshift-image-registry/core/services.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.546335044Z namespaces/openshift-image-registry/discovery.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.546375555Z namespaces/openshift-image-registry/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.546430738Z namespaces/openshift-image-registry/image.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.54647658Z namespaces/openshift-image-registry/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.546518462Z namespaces/openshift-image-registry/k8s.ovn.org/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.546568295Z namespaces/openshift-image-registry/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.546652508Z namespaces/openshift-image-registry/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.546736072Z namespaces/openshift-image-registry/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.546797855Z namespaces/openshift-image-registry/monitoring.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.546845837Z namespaces/openshift-image-registry/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.546911111Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.547032006Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/image-registry-operator-alerts.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.547144551Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/image-registry-rules.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.547236216Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/imagestreams-rules.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.54731585Z namespaces/openshift-image-registry/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.547352691Z namespaces/openshift-image-registry/monitoring.coreos.com/servicemonitors/image-registry.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.547418824Z namespaces/openshift-image-registry/networking.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.547450446Z namespaces/openshift-image-registry/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.547499348Z namespaces/openshift-image-registry/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.547515449Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.547578982Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.54775808Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.54795078Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.548122938Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.548347498Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.548539897Z namespaces/openshift-image-registry/pods/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.548548258Z namespaces/openshift-image-registry/pods/image-registry-86496c8854-sxqfg/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.548571279Z namespaces/openshift-image-registry/pods/image-registry-86496c8854-sxqfg/image-registry-86496c8854-sxqfg.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.548691515Z namespaces/openshift-image-registry/pods/image-registry-86496c8854-sxqfg/registry/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.548701015Z namespaces/openshift-image-registry/pods/image-registry-86496c8854-sxqfg/registry/registry/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.548713196Z namespaces/openshift-image-registry/pods/image-registry-86496c8854-sxqfg/registry/registry/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.548756127Z namespaces/openshift-image-registry/pods/image-registry-86496c8854-sxqfg/registry/registry/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.550964321Z namespaces/openshift-image-registry/pods/image-registry-86496c8854-sxqfg/registry/registry/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.551130909Z namespaces/openshift-image-registry/pods/image-registry-86496c8854-sxqfg/registry/registry/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.55114651Z namespaces/openshift-image-registry/pods/node-ca-6nvz2/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.551203313Z namespaces/openshift-image-registry/pods/node-ca-6nvz2/node-ca-6nvz2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.551297507Z namespaces/openshift-image-registry/pods/node-ca-6nvz2/node-ca/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.551307747Z namespaces/openshift-image-registry/pods/node-ca-6nvz2/node-ca/node-ca/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.551313678Z namespaces/openshift-image-registry/pods/node-ca-6nvz2/node-ca/node-ca/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.551346279Z namespaces/openshift-image-registry/pods/node-ca-6nvz2/node-ca/node-ca/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.551524798Z namespaces/openshift-image-registry/pods/node-ca-6nvz2/node-ca/node-ca/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.551594151Z namespaces/openshift-image-registry/pods/node-ca-6nvz2/node-ca/node-ca/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.551625882Z namespaces/openshift-image-registry/pods/node-ca-8gj9q/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.551687025Z namespaces/openshift-image-registry/pods/node-ca-8gj9q/node-ca-8gj9q.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.551759639Z namespaces/openshift-image-registry/pods/node-ca-8gj9q/node-ca/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.551766759Z namespaces/openshift-image-registry/pods/node-ca-8gj9q/node-ca/node-ca/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.551773549Z namespaces/openshift-image-registry/pods/node-ca-8gj9q/node-ca/node-ca/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.551830832Z namespaces/openshift-image-registry/pods/node-ca-8gj9q/node-ca/node-ca/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.552052202Z namespaces/openshift-image-registry/pods/node-ca-8gj9q/node-ca/node-ca/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.552119346Z namespaces/openshift-image-registry/pods/node-ca-8gj9q/node-ca/node-ca/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.552151587Z namespaces/openshift-image-registry/pods/node-ca-b46vj/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.55220829Z namespaces/openshift-image-registry/pods/node-ca-b46vj/node-ca-b46vj.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.552281353Z namespaces/openshift-image-registry/pods/node-ca-b46vj/node-ca/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.552288864Z namespaces/openshift-image-registry/pods/node-ca-b46vj/node-ca/node-ca/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.552298214Z namespaces/openshift-image-registry/pods/node-ca-b46vj/node-ca/node-ca/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.552351027Z namespaces/openshift-image-registry/pods/node-ca-b46vj/node-ca/node-ca/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.552526605Z namespaces/openshift-image-registry/pods/node-ca-b46vj/node-ca/node-ca/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.552595378Z namespaces/openshift-image-registry/pods/node-ca-b46vj/node-ca/node-ca/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.55262827Z namespaces/openshift-image-registry/policy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.552658761Z namespaces/openshift-image-registry/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.552732714Z namespaces/openshift-image-registry/route.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.552783477Z namespaces/openshift-image-registry/route.openshift.io/routes.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.552830259Z namespaces/openshift-infra/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.552838999Z namespaces/openshift-infra/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.55284395Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.552885372Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.55305657Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.553235128Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.553372625Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.553575184Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.553923141Z namespaces/openshift-ingress-canary/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.553966633Z namespaces/openshift-ingress-canary/openshift-ingress-canary.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.554074838Z namespaces/openshift-ingress-canary/apps.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.554104989Z namespaces/openshift-ingress-canary/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.554168532Z namespaces/openshift-ingress-canary/apps/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.554209774Z namespaces/openshift-ingress-canary/apps/daemonsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.554390463Z namespaces/openshift-ingress-canary/apps/deployments.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.554464476Z namespaces/openshift-ingress-canary/apps/replicasets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.55454889Z namespaces/openshift-ingress-canary/apps/statefulsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.554604483Z namespaces/openshift-ingress-canary/autoscaling/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.554649015Z namespaces/openshift-ingress-canary/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.554712388Z namespaces/openshift-ingress-canary/batch/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.554743209Z namespaces/openshift-ingress-canary/batch/cronjobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.554838144Z namespaces/openshift-ingress-canary/batch/jobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.554867265Z namespaces/openshift-ingress-canary/build.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.554919668Z namespaces/openshift-ingress-canary/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.555017282Z namespaces/openshift-ingress-canary/build.openshift.io/builds.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.555110857Z namespaces/openshift-ingress-canary/core/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.555138368Z namespaces/openshift-ingress-canary/core/configmaps.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.555262273Z namespaces/openshift-ingress-canary/core/endpoints.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.555367309Z namespaces/openshift-ingress-canary/core/events.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.555512275Z namespaces/openshift-ingress-canary/core/persistentvolumeclaims.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.555598479Z namespaces/openshift-ingress-canary/core/pods.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.555774348Z namespaces/openshift-ingress-canary/core/replicationcontrollers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.555901974Z namespaces/openshift-ingress-canary/core/secrets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.556090072Z namespaces/openshift-ingress-canary/core/services.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.556150425Z namespaces/openshift-ingress-canary/discovery.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.556192277Z namespaces/openshift-ingress-canary/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.55624789Z namespaces/openshift-ingress-canary/image.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.556291942Z namespaces/openshift-ingress-canary/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.556361025Z namespaces/openshift-ingress-canary/k8s.ovn.org/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.556407827Z namespaces/openshift-ingress-canary/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.556489471Z namespaces/openshift-ingress-canary/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.556580166Z namespaces/openshift-ingress-canary/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.556631878Z namespaces/openshift-ingress-canary/monitoring.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.55667715Z namespaces/openshift-ingress-canary/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.556723962Z namespaces/openshift-ingress-canary/networking.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.556765854Z namespaces/openshift-ingress-canary/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.556823087Z namespaces/openshift-ingress-canary/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.556836427Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.55689253Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.557060358Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.557248357Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.557384973Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.557592363Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.558332168Z namespaces/openshift-ingress-canary/pods/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.558343858Z namespaces/openshift-ingress-canary/pods/ingress-canary-796bm/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.558355369Z namespaces/openshift-ingress-canary/pods/ingress-canary-796bm/ingress-canary-796bm.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.558453094Z namespaces/openshift-ingress-canary/pods/ingress-canary-796bm/serve-healthcheck-canary/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.558460224Z namespaces/openshift-ingress-canary/pods/ingress-canary-796bm/serve-healthcheck-canary/serve-healthcheck-canary/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.558470815Z namespaces/openshift-ingress-canary/pods/ingress-canary-796bm/serve-healthcheck-canary/serve-healthcheck-canary/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.558514947Z namespaces/openshift-ingress-canary/pods/ingress-canary-796bm/serve-healthcheck-canary/serve-healthcheck-canary/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.558621182Z namespaces/openshift-ingress-canary/pods/ingress-canary-796bm/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.558692445Z namespaces/openshift-ingress-canary/pods/ingress-canary-796bm/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.558744637Z namespaces/openshift-ingress-canary/pods/ingress-canary-8tqhw/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.558777899Z namespaces/openshift-ingress-canary/pods/ingress-canary-8tqhw/ingress-canary-8tqhw.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.558870353Z namespaces/openshift-ingress-canary/pods/ingress-canary-8tqhw/serve-healthcheck-canary/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.558880044Z namespaces/openshift-ingress-canary/pods/ingress-canary-8tqhw/serve-healthcheck-canary/serve-healthcheck-canary/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.558884974Z namespaces/openshift-ingress-canary/pods/ingress-canary-8tqhw/serve-healthcheck-canary/serve-healthcheck-canary/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.558928986Z namespaces/openshift-ingress-canary/pods/ingress-canary-8tqhw/serve-healthcheck-canary/serve-healthcheck-canary/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.559022181Z namespaces/openshift-ingress-canary/pods/ingress-canary-8tqhw/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.559180838Z namespaces/openshift-ingress-canary/pods/ingress-canary-8tqhw/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.55922835Z namespaces/openshift-ingress-canary/pods/ingress-canary-ggdhq/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.559260152Z namespaces/openshift-ingress-canary/pods/ingress-canary-ggdhq/ingress-canary-ggdhq.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.559356056Z namespaces/openshift-ingress-canary/pods/ingress-canary-ggdhq/serve-healthcheck-canary/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.559367937Z namespaces/openshift-ingress-canary/pods/ingress-canary-ggdhq/serve-healthcheck-canary/serve-healthcheck-canary/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.559378717Z namespaces/openshift-ingress-canary/pods/ingress-canary-ggdhq/serve-healthcheck-canary/serve-healthcheck-canary/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.559426709Z namespaces/openshift-ingress-canary/pods/ingress-canary-ggdhq/serve-healthcheck-canary/serve-healthcheck-canary/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.559523404Z namespaces/openshift-ingress-canary/pods/ingress-canary-ggdhq/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.559590417Z namespaces/openshift-ingress-canary/pods/ingress-canary-ggdhq/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.559627049Z namespaces/openshift-ingress-canary/policy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.559667541Z namespaces/openshift-ingress-canary/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.559732124Z namespaces/openshift-ingress-canary/route.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.559776716Z namespaces/openshift-ingress-canary/route.openshift.io/routes.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.559873861Z namespaces/openshift-ingress-operator/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.559929653Z namespaces/openshift-ingress-operator/openshift-ingress-operator.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.559998656Z namespaces/openshift-ingress-operator/apps.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.560063119Z namespaces/openshift-ingress-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.560131673Z namespaces/openshift-ingress-operator/apps/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.560159384Z namespaces/openshift-ingress-operator/apps/daemonsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.560250338Z namespaces/openshift-ingress-operator/apps/deployments.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.560325992Z namespaces/openshift-ingress-operator/apps/replicasets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.560404335Z namespaces/openshift-ingress-operator/apps/statefulsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.560461578Z namespaces/openshift-ingress-operator/autoscaling/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.56049878Z namespaces/openshift-ingress-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.560570743Z namespaces/openshift-ingress-operator/batch/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.560598275Z namespaces/openshift-ingress-operator/batch/cronjobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.560691219Z namespaces/openshift-ingress-operator/batch/jobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.560739491Z namespaces/openshift-ingress-operator/build.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.560781013Z namespaces/openshift-ingress-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.560858007Z namespaces/openshift-ingress-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.56093183Z namespaces/openshift-ingress-operator/core/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.560977562Z namespaces/openshift-ingress-operator/core/configmaps.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.561870504Z namespaces/openshift-ingress-operator/core/endpoints.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.561959099Z namespaces/openshift-ingress-operator/core/events.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.562108826Z namespaces/openshift-ingress-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.562191019Z namespaces/openshift-ingress-operator/core/pods.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.562271643Z namespaces/openshift-ingress-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.562429111Z namespaces/openshift-ingress-operator/core/secrets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.562613739Z namespaces/openshift-ingress-operator/core/services.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.562676622Z namespaces/openshift-ingress-operator/discovery.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.562705964Z namespaces/openshift-ingress-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.562787488Z namespaces/openshift-ingress-operator/image.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.56283654Z namespaces/openshift-ingress-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.562889352Z namespaces/openshift-ingress-operator/ingress.operator.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.562899653Z namespaces/openshift-ingress-operator/ingress.operator.openshift.io/dnsrecords/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.562956735Z namespaces/openshift-ingress-operator/ingress.operator.openshift.io/dnsrecords/default-wildcard.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.563061151Z namespaces/openshift-ingress-operator/k8s.ovn.org/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.563103002Z namespaces/openshift-ingress-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.563192787Z namespaces/openshift-ingress-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.56326869Z namespaces/openshift-ingress-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.563353914Z namespaces/openshift-ingress-operator/monitoring.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.563407357Z namespaces/openshift-ingress-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.56346399Z namespaces/openshift-ingress-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.563517052Z namespaces/openshift-ingress-operator/monitoring.coreos.com/prometheusrules/ingress-operator.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.563590445Z namespaces/openshift-ingress-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.563630667Z namespaces/openshift-ingress-operator/monitoring.coreos.com/servicemonitors/ingress-operator.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.56369719Z namespaces/openshift-ingress-operator/networking.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.563747943Z namespaces/openshift-ingress-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.563802235Z namespaces/openshift-ingress-operator/operator.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.563809506Z namespaces/openshift-ingress-operator/operator.openshift.io/ingresscontrollers/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.563844317Z namespaces/openshift-ingress-operator/operator.openshift.io/ingresscontrollers/default.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.563924161Z namespaces/openshift-ingress-operator/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.563934302Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.563984914Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.564207794Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.564403244Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.56454585Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.564786232Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.565055064Z namespaces/openshift-ingress-operator/policy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.56517584Z namespaces/openshift-ingress-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.565236393Z namespaces/openshift-ingress-operator/route.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.565285495Z namespaces/openshift-ingress-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.56537938Z namespaces/openshift-ingress/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.565440872Z namespaces/openshift-ingress/openshift-ingress.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.565540727Z namespaces/openshift-ingress/apps.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.565588199Z namespaces/openshift-ingress/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.565659943Z namespaces/openshift-ingress/apps/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.565689334Z namespaces/openshift-ingress/apps/daemonsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.565780258Z namespaces/openshift-ingress/apps/deployments.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.56601651Z namespaces/openshift-ingress/apps/replicasets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.566406528Z namespaces/openshift-ingress/apps/statefulsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.56646372Z namespaces/openshift-ingress/autoscaling/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.566515873Z namespaces/openshift-ingress/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.566572916Z namespaces/openshift-ingress/batch/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.566607097Z namespaces/openshift-ingress/batch/cronjobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.566689721Z namespaces/openshift-ingress/batch/jobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.566749584Z namespaces/openshift-ingress/build.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.566795776Z namespaces/openshift-ingress/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.56687667Z namespaces/openshift-ingress/build.openshift.io/builds.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.566933793Z namespaces/openshift-ingress/coordination.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.566947513Z namespaces/openshift-ingress/coordination.k8s.io/leases/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.566994965Z namespaces/openshift-ingress/coordination.k8s.io/leases/istio-gateway-ca-openshift-gateway.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.567172174Z namespaces/openshift-ingress/coordination.k8s.io/leases/istio-gateway-deployment-openshift-gateway.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.567231057Z namespaces/openshift-ingress/coordination.k8s.io/leases/istio-gateway-status-leader-openshift-gateway.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.567347062Z namespaces/openshift-ingress/core/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.567379914Z namespaces/openshift-ingress/core/configmaps.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.567774372Z namespaces/openshift-ingress/core/endpoints.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.567874827Z namespaces/openshift-ingress/core/events.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.56814185Z namespaces/openshift-ingress/core/persistentvolumeclaims.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.568222283Z namespaces/openshift-ingress/core/pods.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.568466475Z namespaces/openshift-ingress/core/replicationcontrollers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.568596521Z namespaces/openshift-ingress/core/secrets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.568804121Z namespaces/openshift-ingress/core/services.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.568897215Z namespaces/openshift-ingress/discovery.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.568945197Z namespaces/openshift-ingress/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.569057583Z namespaces/openshift-ingress/gateway.networking.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.569071693Z namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.569110985Z namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/openshift-ai-inference.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.569183919Z namespaces/openshift-ingress/image.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.56921628Z namespaces/openshift-ingress/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.569290874Z namespaces/openshift-ingress/k8s.ovn.org/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.569333566Z namespaces/openshift-ingress/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.569410239Z namespaces/openshift-ingress/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.569499023Z namespaces/openshift-ingress/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.569553706Z namespaces/openshift-ingress/monitoring.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.569618689Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.569702383Z namespaces/openshift-ingress/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.569765466Z namespaces/openshift-ingress/monitoring.coreos.com/podmonitors/openshift-ai-inference-metrics.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.56986452Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.569921863Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors/router-default.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.569994707Z namespaces/openshift-ingress/networking.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.570104842Z namespaces/openshift-ingress/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.570174775Z namespaces/openshift-ingress/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.570187516Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.570248409Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.570408046Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.570585765Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.570728361Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.57092783Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.571161832Z namespaces/openshift-ingress/pods/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.571172882Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-46d7p/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.571208234Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-46d7p/istiod-openshift-gateway-5677ccb49b-46d7p.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.57134385Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-46d7p/discovery/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.571353971Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-46d7p/discovery/discovery/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.571362891Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-46d7p/discovery/discovery/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.571407733Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-46d7p/discovery/discovery/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.581662096Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-46d7p/discovery/discovery/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.581734789Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-46d7p/discovery/discovery/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.581791382Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-xf9js/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.581842594Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-xf9js/openshift-ai-inference-openshift-default-8699b59bbc-xf9js.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.581952879Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-xf9js/istio-proxy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.58196168Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-xf9js/istio-proxy/istio-proxy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.5819723Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-xf9js/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.582016122Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-xf9js/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.583346805Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-xf9js/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.58345325Z namespaces/openshift-ingress/pods/router-default-d6d96c7dc-qc8qb/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.583510382Z namespaces/openshift-ingress/pods/router-default-d6d96c7dc-qc8qb/router-default-d6d96c7dc-qc8qb.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.583649499Z namespaces/openshift-ingress/pods/router-default-d6d96c7dc-qc8qb/router/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.58365986Z namespaces/openshift-ingress/pods/router-default-d6d96c7dc-qc8qb/router/router/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.58366794Z namespaces/openshift-ingress/pods/router-default-d6d96c7dc-qc8qb/router/router/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.583708182Z namespaces/openshift-ingress/pods/router-default-d6d96c7dc-qc8qb/router/router/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.583822627Z namespaces/openshift-ingress/pods/router-default-d6d96c7dc-qc8qb/router/router/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.58388681Z namespaces/openshift-ingress/pods/router-default-d6d96c7dc-qc8qb/router/router/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.583922692Z namespaces/openshift-ingress/policy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.583967734Z namespaces/openshift-ingress/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.584050988Z namespaces/openshift-ingress/route.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.58409647Z namespaces/openshift-ingress/route.openshift.io/routes.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.584208915Z namespaces/openshift-insights/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.584242117Z namespaces/openshift-insights/openshift-insights.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.584341522Z namespaces/openshift-insights/apps.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.584386704Z namespaces/openshift-insights/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.584470918Z namespaces/openshift-insights/apps/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.584496009Z namespaces/openshift-insights/apps/daemonsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.584624095Z namespaces/openshift-insights/apps/deployments.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.58473311Z namespaces/openshift-insights/apps/replicasets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.584840005Z namespaces/openshift-insights/apps/statefulsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.584886867Z namespaces/openshift-insights/apps/deployments/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.584928459Z namespaces/openshift-insights/apps/deployments/insights-operator.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.585000443Z namespaces/openshift-insights/autoscaling/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.585070136Z namespaces/openshift-insights/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.585130619Z namespaces/openshift-insights/batch/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.585174191Z namespaces/openshift-insights/batch/cronjobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.585274465Z namespaces/openshift-insights/batch/jobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.585334198Z namespaces/openshift-insights/build.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.58538073Z namespaces/openshift-insights/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.585461044Z namespaces/openshift-insights/build.openshift.io/builds.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.585535488Z namespaces/openshift-insights/core/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.58556943Z namespaces/openshift-insights/core/configmaps.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.586564416Z namespaces/openshift-insights/core/endpoints.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.586663311Z namespaces/openshift-insights/core/events.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.586956325Z namespaces/openshift-insights/core/persistentvolumeclaims.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.587108862Z namespaces/openshift-insights/core/pods.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.587433587Z namespaces/openshift-insights/core/replicationcontrollers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.587539672Z namespaces/openshift-insights/core/secrets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.58770146Z namespaces/openshift-insights/core/services.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.587765053Z namespaces/openshift-insights/core/configmaps/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.587793584Z namespaces/openshift-insights/core/configmaps/service-ca-bundle.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.587871918Z namespaces/openshift-insights/core/serviceaccounts/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.587900439Z namespaces/openshift-insights/core/serviceaccounts/gather.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.588018545Z namespaces/openshift-insights/core/serviceaccounts/operator.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.588078317Z namespaces/openshift-insights/core/services/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.58812504Z namespaces/openshift-insights/core/services/metrics.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.588190313Z namespaces/openshift-insights/discovery.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.588239585Z namespaces/openshift-insights/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.588294198Z namespaces/openshift-insights/image.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.58834395Z namespaces/openshift-insights/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.588399432Z namespaces/openshift-insights/k8s.ovn.org/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.588448385Z namespaces/openshift-insights/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.588525678Z namespaces/openshift-insights/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.588604402Z namespaces/openshift-insights/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.588665825Z namespaces/openshift-insights/monitoring.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.588706457Z namespaces/openshift-insights/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.588779421Z namespaces/openshift-insights/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.588806822Z namespaces/openshift-insights/monitoring.coreos.com/prometheusrules/insights-prometheus-rules.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.588890506Z namespaces/openshift-insights/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.588919187Z namespaces/openshift-insights/monitoring.coreos.com/servicemonitors/insights-operator.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.588996591Z namespaces/openshift-insights/networking.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.589055794Z namespaces/openshift-insights/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.589113006Z namespaces/openshift-insights/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.589122877Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.589178929Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.589359378Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.589664682Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.589817749Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.590093212Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.590331454Z namespaces/openshift-insights/pods/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.590341564Z namespaces/openshift-insights/pods/insights-operator-5885fdb759-5dgxd/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.590362995Z namespaces/openshift-insights/pods/insights-operator-5885fdb759-5dgxd/insights-operator-5885fdb759-5dgxd.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.59047169Z namespaces/openshift-insights/pods/insights-operator-5885fdb759-5dgxd/insights-operator/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.590480041Z namespaces/openshift-insights/pods/insights-operator-5885fdb759-5dgxd/insights-operator/insights-operator/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.590486491Z namespaces/openshift-insights/pods/insights-operator-5885fdb759-5dgxd/insights-operator/insights-operator/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.590537433Z namespaces/openshift-insights/pods/insights-operator-5885fdb759-5dgxd/insights-operator/insights-operator/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.590964713Z namespaces/openshift-insights/pods/insights-operator-5885fdb759-5dgxd/insights-operator/insights-operator/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.591444476Z namespaces/openshift-insights/pods/insights-runtime-extractor-2df6f/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.591483788Z namespaces/openshift-insights/pods/insights-runtime-extractor-2df6f/insights-runtime-extractor-2df6f.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.591578122Z namespaces/openshift-insights/pods/insights-runtime-extractor-2df6f/exporter/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.591587193Z namespaces/openshift-insights/pods/insights-runtime-extractor-2df6f/exporter/exporter/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.591591293Z namespaces/openshift-insights/pods/insights-runtime-extractor-2df6f/exporter/exporter/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.591626004Z namespaces/openshift-insights/pods/insights-runtime-extractor-2df6f/exporter/exporter/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.591725069Z namespaces/openshift-insights/pods/insights-runtime-extractor-2df6f/exporter/exporter/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.591804283Z namespaces/openshift-insights/pods/insights-runtime-extractor-2df6f/exporter/exporter/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.591846125Z namespaces/openshift-insights/pods/insights-runtime-extractor-2df6f/extractor/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.591855875Z namespaces/openshift-insights/pods/insights-runtime-extractor-2df6f/extractor/extractor/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.591860815Z namespaces/openshift-insights/pods/insights-runtime-extractor-2df6f/extractor/extractor/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.591890707Z namespaces/openshift-insights/pods/insights-runtime-extractor-2df6f/extractor/extractor/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.591999642Z namespaces/openshift-insights/pods/insights-runtime-extractor-2df6f/extractor/extractor/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.592088416Z namespaces/openshift-insights/pods/insights-runtime-extractor-2df6f/extractor/extractor/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.592105787Z namespaces/openshift-insights/pods/insights-runtime-extractor-2df6f/kube-rbac-proxy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.592140939Z namespaces/openshift-insights/pods/insights-runtime-extractor-2df6f/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.592149229Z namespaces/openshift-insights/pods/insights-runtime-extractor-2df6f/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.592197651Z namespaces/openshift-insights/pods/insights-runtime-extractor-2df6f/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.592292466Z namespaces/openshift-insights/pods/insights-runtime-extractor-2df6f/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.592352569Z namespaces/openshift-insights/pods/insights-runtime-extractor-2df6f/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.592398981Z namespaces/openshift-insights/pods/insights-runtime-extractor-hbvl8/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.592432002Z namespaces/openshift-insights/pods/insights-runtime-extractor-hbvl8/insights-runtime-extractor-hbvl8.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.592526947Z namespaces/openshift-insights/pods/insights-runtime-extractor-hbvl8/exporter/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.592535667Z namespaces/openshift-insights/pods/insights-runtime-extractor-hbvl8/exporter/exporter/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.592543447Z namespaces/openshift-insights/pods/insights-runtime-extractor-hbvl8/exporter/exporter/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.59259021Z namespaces/openshift-insights/pods/insights-runtime-extractor-hbvl8/exporter/exporter/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.592676774Z namespaces/openshift-insights/pods/insights-runtime-extractor-hbvl8/exporter/exporter/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.592745707Z namespaces/openshift-insights/pods/insights-runtime-extractor-hbvl8/exporter/exporter/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.592769778Z namespaces/openshift-insights/pods/insights-runtime-extractor-hbvl8/extractor/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.592793259Z namespaces/openshift-insights/pods/insights-runtime-extractor-hbvl8/extractor/extractor/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.59279959Z namespaces/openshift-insights/pods/insights-runtime-extractor-hbvl8/extractor/extractor/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.592843612Z namespaces/openshift-insights/pods/insights-runtime-extractor-hbvl8/extractor/extractor/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.592942136Z namespaces/openshift-insights/pods/insights-runtime-extractor-hbvl8/extractor/extractor/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.59302762Z namespaces/openshift-insights/pods/insights-runtime-extractor-hbvl8/extractor/extractor/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.593072462Z namespaces/openshift-insights/pods/insights-runtime-extractor-hbvl8/kube-rbac-proxy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.593109514Z namespaces/openshift-insights/pods/insights-runtime-extractor-hbvl8/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.593121405Z namespaces/openshift-insights/pods/insights-runtime-extractor-hbvl8/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.593155096Z namespaces/openshift-insights/pods/insights-runtime-extractor-hbvl8/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.593265892Z namespaces/openshift-insights/pods/insights-runtime-extractor-hbvl8/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.593367686Z namespaces/openshift-insights/pods/insights-runtime-extractor-hbvl8/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.593424679Z namespaces/openshift-insights/pods/insights-runtime-extractor-q6pkk/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.593487402Z namespaces/openshift-insights/pods/insights-runtime-extractor-q6pkk/insights-runtime-extractor-q6pkk.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.593594497Z namespaces/openshift-insights/pods/insights-runtime-extractor-q6pkk/exporter/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.593602897Z namespaces/openshift-insights/pods/insights-runtime-extractor-q6pkk/exporter/exporter/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.593613558Z namespaces/openshift-insights/pods/insights-runtime-extractor-q6pkk/exporter/exporter/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.59365826Z namespaces/openshift-insights/pods/insights-runtime-extractor-q6pkk/exporter/exporter/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.593746704Z namespaces/openshift-insights/pods/insights-runtime-extractor-q6pkk/exporter/exporter/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.593816767Z namespaces/openshift-insights/pods/insights-runtime-extractor-q6pkk/exporter/exporter/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.593853629Z namespaces/openshift-insights/pods/insights-runtime-extractor-q6pkk/extractor/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.59386078Z namespaces/openshift-insights/pods/insights-runtime-extractor-q6pkk/extractor/extractor/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.59386479Z namespaces/openshift-insights/pods/insights-runtime-extractor-q6pkk/extractor/extractor/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.593901851Z namespaces/openshift-insights/pods/insights-runtime-extractor-q6pkk/extractor/extractor/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.593997366Z namespaces/openshift-insights/pods/insights-runtime-extractor-q6pkk/extractor/extractor/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.59408946Z namespaces/openshift-insights/pods/insights-runtime-extractor-q6pkk/extractor/extractor/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.594127092Z namespaces/openshift-insights/pods/insights-runtime-extractor-q6pkk/kube-rbac-proxy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.594139843Z namespaces/openshift-insights/pods/insights-runtime-extractor-q6pkk/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.594145023Z namespaces/openshift-insights/pods/insights-runtime-extractor-q6pkk/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.594158263Z namespaces/openshift-insights/pods/insights-runtime-extractor-q6pkk/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.594261818Z namespaces/openshift-insights/pods/insights-runtime-extractor-q6pkk/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.594368404Z namespaces/openshift-insights/pods/insights-runtime-extractor-q6pkk/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.594387454Z namespaces/openshift-insights/policy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.594440377Z namespaces/openshift-insights/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.59451423Z namespaces/openshift-insights/route.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.594568243Z namespaces/openshift-insights/route.openshift.io/routes.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.594644257Z namespaces/openshift-keda/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.594651297Z namespaces/openshift-keda/coordination.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.594655437Z namespaces/openshift-keda/coordination.k8s.io/leases/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.594702379Z namespaces/openshift-keda/coordination.k8s.io/leases/olm-operator.keda.sh.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.594796974Z namespaces/openshift-keda/coordination.k8s.io/leases/operator.keda.sh.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.594845926Z namespaces/openshift-keda/monitoring.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.594855696Z namespaces/openshift-keda/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.594886318Z namespaces/openshift-keda/monitoring.coreos.com/podmonitors/keda-olm-operator.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.594954761Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.595020034Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/keda-admission-webhooks.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.5951525Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/keda-metrics-apiserver.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.595237654Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/keda-operator.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.595301327Z namespaces/openshift-keda/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.595312378Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.59536652Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.595518527Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.595693066Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/custom-metrics-autoscaler.v2.19.0-2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.595897125Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.596142157Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.596292414Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.596462372Z namespaces/openshift-keda/operators.coreos.com/installplans/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.596501034Z namespaces/openshift-keda/operators.coreos.com/installplans/install-bl8w7.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.596657111Z namespaces/openshift-keda/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.596713944Z namespaces/openshift-keda/operators.coreos.com/operatorconditions/custom-metrics-autoscaler.v2.19.0-2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.596778097Z namespaces/openshift-keda/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.596812028Z namespaces/openshift-keda/operators.coreos.com/operatorgroups/openshift-keda.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.596873781Z namespaces/openshift-keda/operators.coreos.com/subscriptions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.596915993Z namespaces/openshift-keda/operators.coreos.com/subscriptions/openshift-custom-metrics-autoscaler-operator.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.596993517Z namespaces/openshift-kube-apiserver-operator/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.59706436Z namespaces/openshift-kube-apiserver-operator/openshift-kube-apiserver-operator.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.597122513Z namespaces/openshift-kube-apiserver-operator/apps.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.597152044Z namespaces/openshift-kube-apiserver-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.597218238Z namespaces/openshift-kube-apiserver-operator/apps/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.597248519Z namespaces/openshift-kube-apiserver-operator/apps/daemonsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.597337123Z namespaces/openshift-kube-apiserver-operator/apps/deployments.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.597435718Z namespaces/openshift-kube-apiserver-operator/apps/replicasets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.597515341Z namespaces/openshift-kube-apiserver-operator/apps/statefulsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.597561824Z namespaces/openshift-kube-apiserver-operator/autoscaling/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.597595975Z namespaces/openshift-kube-apiserver-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.597649918Z namespaces/openshift-kube-apiserver-operator/batch/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.597679269Z namespaces/openshift-kube-apiserver-operator/batch/cronjobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.597769433Z namespaces/openshift-kube-apiserver-operator/batch/jobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.597819286Z namespaces/openshift-kube-apiserver-operator/build.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.597848007Z namespaces/openshift-kube-apiserver-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.597940632Z namespaces/openshift-kube-apiserver-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.597993174Z namespaces/openshift-kube-apiserver-operator/core/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.598047537Z namespaces/openshift-kube-apiserver-operator/core/configmaps.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.598162762Z namespaces/openshift-kube-apiserver-operator/core/endpoints.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.598237666Z namespaces/openshift-kube-apiserver-operator/core/events.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.598391013Z namespaces/openshift-kube-apiserver-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.598473717Z namespaces/openshift-kube-apiserver-operator/core/pods.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.59855208Z namespaces/openshift-kube-apiserver-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.598670876Z namespaces/openshift-kube-apiserver-operator/core/secrets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.598877705Z namespaces/openshift-kube-apiserver-operator/core/services.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.598936228Z namespaces/openshift-kube-apiserver-operator/discovery.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.59896527Z namespaces/openshift-kube-apiserver-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.599032663Z namespaces/openshift-kube-apiserver-operator/image.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.599097956Z namespaces/openshift-kube-apiserver-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.599153829Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.599197421Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.599272414Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.599379609Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.599453683Z namespaces/openshift-kube-apiserver-operator/monitoring.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.599502755Z namespaces/openshift-kube-apiserver-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.599551817Z namespaces/openshift-kube-apiserver-operator/networking.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.599580229Z namespaces/openshift-kube-apiserver-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.599640741Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.599649472Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.599695554Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.599856482Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.600061801Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.600193118Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.600396937Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.600585436Z namespaces/openshift-kube-apiserver-operator/policy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.600614707Z namespaces/openshift-kube-apiserver-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.60067473Z namespaces/openshift-kube-apiserver-operator/route.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.600703311Z namespaces/openshift-kube-apiserver-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.600776445Z namespaces/openshift-kube-apiserver/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.6008899Z namespaces/openshift-kube-apiserver/openshift-kube-apiserver.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.600944643Z namespaces/openshift-kube-apiserver/apps.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.600987265Z namespaces/openshift-kube-apiserver/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.601068489Z namespaces/openshift-kube-apiserver/apps/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.60110108Z namespaces/openshift-kube-apiserver/apps/daemonsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.601194995Z namespaces/openshift-kube-apiserver/apps/deployments.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.601271438Z namespaces/openshift-kube-apiserver/apps/replicasets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.601350372Z namespaces/openshift-kube-apiserver/apps/statefulsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.601400534Z namespaces/openshift-kube-apiserver/autoscaling/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.601428676Z namespaces/openshift-kube-apiserver/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.601487739Z namespaces/openshift-kube-apiserver/batch/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.60151633Z namespaces/openshift-kube-apiserver/batch/cronjobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.601606794Z namespaces/openshift-kube-apiserver/batch/jobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.601655766Z namespaces/openshift-kube-apiserver/build.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.601685438Z namespaces/openshift-kube-apiserver/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.601781992Z namespaces/openshift-kube-apiserver/build.openshift.io/builds.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.601844315Z namespaces/openshift-kube-apiserver/core/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.601894898Z namespaces/openshift-kube-apiserver/core/configmaps.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.601990352Z namespaces/openshift-kube-apiserver/core/endpoints.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.602089667Z namespaces/openshift-kube-apiserver/core/events.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.60216406Z namespaces/openshift-kube-apiserver/core/persistentvolumeclaims.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.602243824Z namespaces/openshift-kube-apiserver/core/pods.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.602323948Z namespaces/openshift-kube-apiserver/core/replicationcontrollers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.602441963Z namespaces/openshift-kube-apiserver/core/secrets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.602600401Z namespaces/openshift-kube-apiserver/core/services.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.602654593Z namespaces/openshift-kube-apiserver/discovery.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.602700836Z namespaces/openshift-kube-apiserver/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.602747108Z namespaces/openshift-kube-apiserver/image.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.60279057Z namespaces/openshift-kube-apiserver/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.602839942Z namespaces/openshift-kube-apiserver/k8s.ovn.org/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.602881044Z namespaces/openshift-kube-apiserver/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.602961758Z namespaces/openshift-kube-apiserver/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.603068693Z namespaces/openshift-kube-apiserver/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.603107535Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.603162867Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.60322857Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.603267472Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/prometheusrules/api-usage.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.603381657Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/prometheusrules/podsecurity.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.60343933Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.603473672Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/servicemonitors/openshift-kube-apiserver.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.603533405Z namespaces/openshift-kube-apiserver/networking.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.603574266Z namespaces/openshift-kube-apiserver/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.603621279Z namespaces/openshift-kube-apiserver/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.603628069Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.603675541Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.603821798Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.603998417Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.604160344Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.604353083Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.604543852Z namespaces/openshift-kube-apiserver/policy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.604572304Z namespaces/openshift-kube-apiserver/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.604631096Z namespaces/openshift-kube-apiserver/route.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.604675418Z namespaces/openshift-kube-apiserver/route.openshift.io/routes.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.604747922Z namespaces/openshift-kube-controller-manager-operator/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.604782434Z namespaces/openshift-kube-controller-manager-operator/openshift-kube-controller-manager-operator.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.604849907Z namespaces/openshift-kube-controller-manager-operator/apps.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.604884738Z namespaces/openshift-kube-controller-manager-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.604941281Z namespaces/openshift-kube-controller-manager-operator/apps/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.604976843Z namespaces/openshift-kube-controller-manager-operator/apps/daemonsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.605091918Z namespaces/openshift-kube-controller-manager-operator/apps/deployments.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.605161361Z namespaces/openshift-kube-controller-manager-operator/apps/replicasets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.605239135Z namespaces/openshift-kube-controller-manager-operator/apps/statefulsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.605283197Z namespaces/openshift-kube-controller-manager-operator/autoscaling/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.605315739Z namespaces/openshift-kube-controller-manager-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.605375861Z namespaces/openshift-kube-controller-manager-operator/batch/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.605430704Z namespaces/openshift-kube-controller-manager-operator/batch/cronjobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.605526918Z namespaces/openshift-kube-controller-manager-operator/batch/jobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.60557036Z namespaces/openshift-kube-controller-manager-operator/build.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.605611433Z namespaces/openshift-kube-controller-manager-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.605695846Z namespaces/openshift-kube-controller-manager-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.60577435Z namespaces/openshift-kube-controller-manager-operator/core/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.605808152Z namespaces/openshift-kube-controller-manager-operator/core/configmaps.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.605921547Z namespaces/openshift-kube-controller-manager-operator/core/endpoints.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.606019702Z namespaces/openshift-kube-controller-manager-operator/core/events.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.606109676Z namespaces/openshift-kube-controller-manager-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.60618484Z namespaces/openshift-kube-controller-manager-operator/core/pods.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.606265373Z namespaces/openshift-kube-controller-manager-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.606392229Z namespaces/openshift-kube-controller-manager-operator/core/secrets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.606569258Z namespaces/openshift-kube-controller-manager-operator/core/services.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.606638461Z namespaces/openshift-kube-controller-manager-operator/discovery.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.606685833Z namespaces/openshift-kube-controller-manager-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.606734975Z namespaces/openshift-kube-controller-manager-operator/image.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.606768387Z namespaces/openshift-kube-controller-manager-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.60683239Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.606865701Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.606956456Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.607115663Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.607174736Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.607222078Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.607288941Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.607343964Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/servicemonitors/kube-controller-manager-operator.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.607407637Z namespaces/openshift-kube-controller-manager-operator/networking.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.607446559Z namespaces/openshift-kube-controller-manager-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.607488751Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.607514522Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.607567574Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.607718652Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.60789616Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.608058888Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.608252317Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.608434495Z namespaces/openshift-kube-controller-manager-operator/policy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.608472967Z namespaces/openshift-kube-controller-manager-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.60853629Z namespaces/openshift-kube-controller-manager-operator/route.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.608577842Z namespaces/openshift-kube-controller-manager-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.608687587Z namespaces/openshift-kube-controller-manager/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.608722709Z namespaces/openshift-kube-controller-manager/openshift-kube-controller-manager.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.608831844Z namespaces/openshift-kube-controller-manager/apps.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.608879196Z namespaces/openshift-kube-controller-manager/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.608935679Z namespaces/openshift-kube-controller-manager/apps/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.608976561Z namespaces/openshift-kube-controller-manager/apps/daemonsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.609096236Z namespaces/openshift-kube-controller-manager/apps/deployments.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.60916974Z namespaces/openshift-kube-controller-manager/apps/replicasets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.609258184Z namespaces/openshift-kube-controller-manager/apps/statefulsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.609321987Z namespaces/openshift-kube-controller-manager/autoscaling/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.60937495Z namespaces/openshift-kube-controller-manager/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.609430282Z namespaces/openshift-kube-controller-manager/batch/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.609464714Z namespaces/openshift-kube-controller-manager/batch/cronjobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.609575529Z namespaces/openshift-kube-controller-manager/batch/jobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.609631902Z namespaces/openshift-kube-controller-manager/build.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.609680014Z namespaces/openshift-kube-controller-manager/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.609771018Z namespaces/openshift-kube-controller-manager/build.openshift.io/builds.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.609830761Z namespaces/openshift-kube-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.609840401Z namespaces/openshift-kube-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.609871493Z namespaces/openshift-kube-controller-manager/coordination.k8s.io/leases/cluster-policy-controller-lock.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.609969387Z namespaces/openshift-kube-controller-manager/core/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.61002228Z namespaces/openshift-kube-controller-manager/core/configmaps.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.610133365Z namespaces/openshift-kube-controller-manager/core/endpoints.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.610218559Z namespaces/openshift-kube-controller-manager/core/events.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.610651059Z namespaces/openshift-kube-controller-manager/core/persistentvolumeclaims.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.610772795Z namespaces/openshift-kube-controller-manager/core/pods.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.61088199Z namespaces/openshift-kube-controller-manager/core/replicationcontrollers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.611033288Z namespaces/openshift-kube-controller-manager/core/secrets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.611206476Z namespaces/openshift-kube-controller-manager/core/services.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.611256258Z namespaces/openshift-kube-controller-manager/discovery.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.611314541Z namespaces/openshift-kube-controller-manager/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.611348502Z namespaces/openshift-kube-controller-manager/image.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.611430646Z namespaces/openshift-kube-controller-manager/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.611475748Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.61151763Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.611619555Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.611706249Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.61172991Z namespaces/openshift-kube-controller-manager/monitoring.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.611791073Z namespaces/openshift-kube-controller-manager/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.611818774Z namespaces/openshift-kube-controller-manager/networking.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.611872267Z namespaces/openshift-kube-controller-manager/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.611894788Z namespaces/openshift-kube-controller-manager/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.611902158Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.611972692Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.612418193Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.612599711Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.612730317Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.612952188Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.613160948Z namespaces/openshift-kube-controller-manager/policy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.613179658Z namespaces/openshift-kube-controller-manager/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.613250882Z namespaces/openshift-kube-controller-manager/route.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.613286314Z namespaces/openshift-kube-controller-manager/route.openshift.io/routes.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.613351297Z namespaces/openshift-kube-scheduler-operator/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.613392368Z namespaces/openshift-kube-scheduler-operator/openshift-kube-scheduler-operator.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.613461592Z namespaces/openshift-kube-scheduler-operator/apps.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.613489683Z namespaces/openshift-kube-scheduler-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.613554196Z namespaces/openshift-kube-scheduler-operator/apps/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.613582168Z namespaces/openshift-kube-scheduler-operator/apps/daemonsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.613685872Z namespaces/openshift-kube-scheduler-operator/apps/deployments.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.613765976Z namespaces/openshift-kube-scheduler-operator/apps/replicasets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.614440558Z namespaces/openshift-kube-scheduler-operator/apps/statefulsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.61448529Z namespaces/openshift-kube-scheduler-operator/autoscaling/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.614530162Z namespaces/openshift-kube-scheduler-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.614562334Z namespaces/openshift-kube-scheduler-operator/batch/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.614610476Z namespaces/openshift-kube-scheduler-operator/batch/cronjobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.6146909Z namespaces/openshift-kube-scheduler-operator/batch/jobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.614738302Z namespaces/openshift-kube-scheduler-operator/build.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.614771313Z namespaces/openshift-kube-scheduler-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.614854527Z namespaces/openshift-kube-scheduler-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.61491294Z namespaces/openshift-kube-scheduler-operator/core/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.614952512Z namespaces/openshift-kube-scheduler-operator/core/configmaps.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.615092199Z namespaces/openshift-kube-scheduler-operator/core/endpoints.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.615165202Z namespaces/openshift-kube-scheduler-operator/core/events.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.615252486Z namespaces/openshift-kube-scheduler-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.615358721Z namespaces/openshift-kube-scheduler-operator/core/pods.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.615436745Z namespaces/openshift-kube-scheduler-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.615564351Z namespaces/openshift-kube-scheduler-operator/core/secrets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.615730649Z namespaces/openshift-kube-scheduler-operator/core/services.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.615782151Z namespaces/openshift-kube-scheduler-operator/discovery.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.615819663Z namespaces/openshift-kube-scheduler-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.615875485Z namespaces/openshift-kube-scheduler-operator/image.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.615916817Z namespaces/openshift-kube-scheduler-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.61596978Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.615998801Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.616202391Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.616281635Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.616328857Z namespaces/openshift-kube-scheduler-operator/monitoring.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.616370079Z namespaces/openshift-kube-scheduler-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.61640127Z namespaces/openshift-kube-scheduler-operator/networking.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.616524156Z namespaces/openshift-kube-scheduler-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.616570858Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.616578518Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.616629591Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.616776718Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.616955086Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.617136325Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.617345534Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.617533693Z namespaces/openshift-kube-scheduler-operator/policy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.617569705Z namespaces/openshift-kube-scheduler-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.617627948Z namespaces/openshift-kube-scheduler-operator/route.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.617654369Z namespaces/openshift-kube-scheduler-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.617728612Z namespaces/openshift-kube-scheduler/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.617759774Z namespaces/openshift-kube-scheduler/openshift-kube-scheduler.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.617827097Z namespaces/openshift-kube-scheduler/apps.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.617853898Z namespaces/openshift-kube-scheduler/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.617908271Z namespaces/openshift-kube-scheduler/apps/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.617941413Z namespaces/openshift-kube-scheduler/apps/daemonsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.618033007Z namespaces/openshift-kube-scheduler/apps/deployments.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.618136542Z namespaces/openshift-kube-scheduler/apps/replicasets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.618212935Z namespaces/openshift-kube-scheduler/apps/statefulsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.618261368Z namespaces/openshift-kube-scheduler/autoscaling/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.618292199Z namespaces/openshift-kube-scheduler/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.618353112Z namespaces/openshift-kube-scheduler/batch/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.618392964Z namespaces/openshift-kube-scheduler/batch/cronjobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.618477698Z namespaces/openshift-kube-scheduler/batch/jobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.61852356Z namespaces/openshift-kube-scheduler/build.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.618548971Z namespaces/openshift-kube-scheduler/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.618639935Z namespaces/openshift-kube-scheduler/build.openshift.io/builds.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.618692978Z namespaces/openshift-kube-scheduler/core/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.61873744Z namespaces/openshift-kube-scheduler/core/configmaps.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.618832764Z namespaces/openshift-kube-scheduler/core/endpoints.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.618910078Z namespaces/openshift-kube-scheduler/core/events.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.618989932Z namespaces/openshift-kube-scheduler/core/persistentvolumeclaims.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.619093057Z namespaces/openshift-kube-scheduler/core/pods.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.619172731Z namespaces/openshift-kube-scheduler/core/replicationcontrollers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.619299006Z namespaces/openshift-kube-scheduler/core/secrets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.619468674Z namespaces/openshift-kube-scheduler/core/services.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.619523767Z namespaces/openshift-kube-scheduler/discovery.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.619569529Z namespaces/openshift-kube-scheduler/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.619617031Z namespaces/openshift-kube-scheduler/image.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.619657573Z namespaces/openshift-kube-scheduler/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.619714646Z namespaces/openshift-kube-scheduler/k8s.ovn.org/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.619758868Z namespaces/openshift-kube-scheduler/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.619836872Z namespaces/openshift-kube-scheduler/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.619916715Z namespaces/openshift-kube-scheduler/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.619959648Z namespaces/openshift-kube-scheduler/monitoring.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.62001774Z namespaces/openshift-kube-scheduler/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.620068533Z namespaces/openshift-kube-scheduler/networking.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.620104234Z namespaces/openshift-kube-scheduler/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.620157057Z namespaces/openshift-kube-scheduler/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.620165657Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.620208219Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.620352326Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.620534384Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.620672611Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.62087085Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.62108264Z namespaces/openshift-kube-scheduler/policy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.621202916Z namespaces/openshift-kube-scheduler/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.621236777Z namespaces/openshift-kube-scheduler/route.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.62128938Z namespaces/openshift-kube-scheduler/route.openshift.io/routes.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.621361363Z namespaces/openshift-kube-storage-version-migrator-operator/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.621413836Z namespaces/openshift-kube-storage-version-migrator-operator/openshift-kube-storage-version-migrator-operator.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.621475579Z namespaces/openshift-kube-storage-version-migrator-operator/apps.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.62150413Z namespaces/openshift-kube-storage-version-migrator-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.621569493Z namespaces/openshift-kube-storage-version-migrator-operator/apps/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.621605405Z namespaces/openshift-kube-storage-version-migrator-operator/apps/daemonsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.621673308Z namespaces/openshift-kube-storage-version-migrator-operator/apps/deployments.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.621804004Z namespaces/openshift-kube-storage-version-migrator-operator/apps/replicasets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.621904869Z namespaces/openshift-kube-storage-version-migrator-operator/apps/statefulsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.621945081Z namespaces/openshift-kube-storage-version-migrator-operator/autoscaling/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.621992623Z namespaces/openshift-kube-storage-version-migrator-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.622072267Z namespaces/openshift-kube-storage-version-migrator-operator/batch/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.622091768Z namespaces/openshift-kube-storage-version-migrator-operator/batch/cronjobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.622193382Z namespaces/openshift-kube-storage-version-migrator-operator/batch/jobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.622242385Z namespaces/openshift-kube-storage-version-migrator-operator/build.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.622272996Z namespaces/openshift-kube-storage-version-migrator-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.62234886Z namespaces/openshift-kube-storage-version-migrator-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.622403132Z namespaces/openshift-kube-storage-version-migrator-operator/coordination.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.622414193Z namespaces/openshift-kube-storage-version-migrator-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.622439854Z namespaces/openshift-kube-storage-version-migrator-operator/coordination.k8s.io/leases/openshift-kube-storage-version-migrator-operator-lock.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.622507917Z namespaces/openshift-kube-storage-version-migrator-operator/core/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.622555419Z namespaces/openshift-kube-storage-version-migrator-operator/core/configmaps.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.622667245Z namespaces/openshift-kube-storage-version-migrator-operator/core/endpoints.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.622754499Z namespaces/openshift-kube-storage-version-migrator-operator/core/events.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.622928307Z namespaces/openshift-kube-storage-version-migrator-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.623019092Z namespaces/openshift-kube-storage-version-migrator-operator/core/pods.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.623228091Z namespaces/openshift-kube-storage-version-migrator-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.623362437Z namespaces/openshift-kube-storage-version-migrator-operator/core/secrets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.623554997Z namespaces/openshift-kube-storage-version-migrator-operator/core/services.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.62361714Z namespaces/openshift-kube-storage-version-migrator-operator/discovery.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.623660422Z namespaces/openshift-kube-storage-version-migrator-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.623717624Z namespaces/openshift-kube-storage-version-migrator-operator/image.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.623756596Z namespaces/openshift-kube-storage-version-migrator-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.623817259Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.623857411Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.623938425Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.624022129Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.624089092Z namespaces/openshift-kube-storage-version-migrator-operator/monitoring.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.624122473Z namespaces/openshift-kube-storage-version-migrator-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.624181006Z namespaces/openshift-kube-storage-version-migrator-operator/networking.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.624223718Z namespaces/openshift-kube-storage-version-migrator-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.624273951Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.624281181Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.624326573Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.62448693Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.62468135Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.624818846Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.625090159Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.625290768Z namespaces/openshift-kube-storage-version-migrator-operator/pods/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.625298879Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6f4d6fb6bc-wlmvb/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.62533615Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6f4d6fb6bc-wlmvb/kube-storage-version-migrator-operator-6f4d6fb6bc-wlmvb.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.625477677Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6f4d6fb6bc-wlmvb/kube-storage-version-migrator-operator/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.625487637Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6f4d6fb6bc-wlmvb/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.625495258Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6f4d6fb6bc-wlmvb/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.62554119Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6f4d6fb6bc-wlmvb/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.625663496Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6f4d6fb6bc-wlmvb/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.625787302Z namespaces/openshift-kube-storage-version-migrator-operator/policy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.625819773Z namespaces/openshift-kube-storage-version-migrator-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.625854355Z namespaces/openshift-kube-storage-version-migrator-operator/route.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.625911688Z namespaces/openshift-kube-storage-version-migrator-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.62596388Z namespaces/openshift-kube-storage-version-migrator/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.626021013Z namespaces/openshift-kube-storage-version-migrator/openshift-kube-storage-version-migrator.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.626082116Z namespaces/openshift-kube-storage-version-migrator/apps.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.626118857Z namespaces/openshift-kube-storage-version-migrator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.62617763Z namespaces/openshift-kube-storage-version-migrator/apps/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.626221672Z namespaces/openshift-kube-storage-version-migrator/apps/daemonsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.626324237Z namespaces/openshift-kube-storage-version-migrator/apps/deployments.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.626439302Z namespaces/openshift-kube-storage-version-migrator/apps/replicasets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.626526076Z namespaces/openshift-kube-storage-version-migrator/apps/statefulsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.626580479Z namespaces/openshift-kube-storage-version-migrator/autoscaling/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.626621031Z namespaces/openshift-kube-storage-version-migrator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.626671053Z namespaces/openshift-kube-storage-version-migrator/batch/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.626717395Z namespaces/openshift-kube-storage-version-migrator/batch/cronjobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.626792499Z namespaces/openshift-kube-storage-version-migrator/batch/jobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.626846252Z namespaces/openshift-kube-storage-version-migrator/build.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.626889634Z namespaces/openshift-kube-storage-version-migrator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.626967567Z namespaces/openshift-kube-storage-version-migrator/build.openshift.io/builds.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.62701945Z namespaces/openshift-kube-storage-version-migrator/core/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.627078002Z namespaces/openshift-kube-storage-version-migrator/core/configmaps.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.627176117Z namespaces/openshift-kube-storage-version-migrator/core/endpoints.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.627255041Z namespaces/openshift-kube-storage-version-migrator/core/events.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.627396107Z namespaces/openshift-kube-storage-version-migrator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.627471041Z namespaces/openshift-kube-storage-version-migrator/core/pods.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.627582796Z namespaces/openshift-kube-storage-version-migrator/core/replicationcontrollers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.627700962Z namespaces/openshift-kube-storage-version-migrator/core/secrets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.627894711Z namespaces/openshift-kube-storage-version-migrator/core/services.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.627953133Z namespaces/openshift-kube-storage-version-migrator/discovery.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.627981415Z namespaces/openshift-kube-storage-version-migrator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.628063169Z namespaces/openshift-kube-storage-version-migrator/image.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.628111531Z namespaces/openshift-kube-storage-version-migrator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.628155443Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.628209716Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.6282945Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.628405875Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.628467668Z namespaces/openshift-kube-storage-version-migrator/monitoring.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.628528681Z namespaces/openshift-kube-storage-version-migrator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.628583923Z namespaces/openshift-kube-storage-version-migrator/networking.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.628636836Z namespaces/openshift-kube-storage-version-migrator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.628681008Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.628690028Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.62872897Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.628883517Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.629085837Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.629223113Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.629419693Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.629612172Z namespaces/openshift-kube-storage-version-migrator/pods/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.629620662Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-b779r/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.629664874Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-b779r/migrator-6865c9967-b779r.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.629745108Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-b779r/graceful-termination/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.629752298Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-b779r/graceful-termination/graceful-termination/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.629759339Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-b779r/graceful-termination/graceful-termination/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.62980034Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-b779r/graceful-termination/graceful-termination/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.629891085Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-b779r/graceful-termination/graceful-termination/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.629960908Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-b779r/graceful-termination/graceful-termination/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.63000161Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-b779r/migrator/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.630027771Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-b779r/migrator/migrator/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.630032202Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-b779r/migrator/migrator/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.630055043Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-b779r/migrator/migrator/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.630162068Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-b779r/migrator/migrator/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.630231291Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-b779r/migrator/migrator/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.630269333Z namespaces/openshift-kube-storage-version-migrator/policy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.630296234Z namespaces/openshift-kube-storage-version-migrator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.630350356Z namespaces/openshift-kube-storage-version-migrator/route.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.630389818Z namespaces/openshift-kube-storage-version-migrator/route.openshift.io/routes.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.630458442Z namespaces/openshift-lws-operator/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.630504734Z namespaces/openshift-lws-operator/openshift-lws-operator.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.630563916Z namespaces/openshift-lws-operator/apps.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.630614409Z namespaces/openshift-lws-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.630695193Z namespaces/openshift-lws-operator/apps/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.630755065Z namespaces/openshift-lws-operator/apps/daemonsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.63086333Z namespaces/openshift-lws-operator/apps/deployments.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.631027738Z namespaces/openshift-lws-operator/apps/replicasets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.631234098Z namespaces/openshift-lws-operator/apps/statefulsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.631294381Z namespaces/openshift-lws-operator/autoscaling/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.631338293Z namespaces/openshift-lws-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.631392025Z namespaces/openshift-lws-operator/batch/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.631437217Z namespaces/openshift-lws-operator/batch/cronjobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.631516271Z namespaces/openshift-lws-operator/batch/jobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.631563073Z namespaces/openshift-lws-operator/build.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.631602165Z namespaces/openshift-lws-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.631879508Z namespaces/openshift-lws-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.63192386Z namespaces/openshift-lws-operator/coordination.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.631932071Z namespaces/openshift-lws-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.631969893Z namespaces/openshift-lws-operator/coordination.k8s.io/leases/b8b2488c.x-k8s.io.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.632070217Z namespaces/openshift-lws-operator/coordination.k8s.io/leases/openshift-lws-operator-lock.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.632139201Z namespaces/openshift-lws-operator/core/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.632177592Z namespaces/openshift-lws-operator/core/configmaps.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.632287588Z namespaces/openshift-lws-operator/core/endpoints.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.632376352Z namespaces/openshift-lws-operator/core/events.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.632693907Z namespaces/openshift-lws-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.63277152Z namespaces/openshift-lws-operator/core/pods.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.63297673Z namespaces/openshift-lws-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.633120527Z namespaces/openshift-lws-operator/core/secrets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.633301935Z namespaces/openshift-lws-operator/core/services.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.633369508Z namespaces/openshift-lws-operator/discovery.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.633415891Z namespaces/openshift-lws-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.633479253Z namespaces/openshift-lws-operator/image.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.633524266Z namespaces/openshift-lws-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.633567258Z namespaces/openshift-lws-operator/k8s.ovn.org/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.633653412Z namespaces/openshift-lws-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.633736586Z namespaces/openshift-lws-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.633816489Z namespaces/openshift-lws-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.633875812Z namespaces/openshift-lws-operator/monitoring.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.633920704Z namespaces/openshift-lws-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.633980287Z namespaces/openshift-lws-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.634023859Z namespaces/openshift-lws-operator/monitoring.coreos.com/servicemonitors/lws-controller-manager-metrics-monitor.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.634106963Z namespaces/openshift-lws-operator/networking.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.634146925Z namespaces/openshift-lws-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.634198378Z namespaces/openshift-lws-operator/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.634207948Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.63424842Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.634425888Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.634618357Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.634743163Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/leader-worker-set.v1.0.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.634973294Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.635154723Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.635363992Z namespaces/openshift-lws-operator/operators.coreos.com/installplans/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.635434205Z namespaces/openshift-lws-operator/operators.coreos.com/installplans/install-jkdhc.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.63553701Z namespaces/openshift-lws-operator/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.635602373Z namespaces/openshift-lws-operator/operators.coreos.com/operatorconditions/leader-worker-set.v1.0.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.635665507Z namespaces/openshift-lws-operator/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.635697278Z namespaces/openshift-lws-operator/operators.coreos.com/operatorgroups/leader-worker-set.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.635757011Z namespaces/openshift-lws-operator/operators.coreos.com/subscriptions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.635799053Z namespaces/openshift-lws-operator/operators.coreos.com/subscriptions/leader-worker-set.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.635869316Z namespaces/openshift-lws-operator/pods/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.635878537Z namespaces/openshift-lws-operator/pods/lws-controller-manager-5c847d9c7c-9t4vb/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.635906258Z namespaces/openshift-lws-operator/pods/lws-controller-manager-5c847d9c7c-9t4vb/lws-controller-manager-5c847d9c7c-9t4vb.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.636001122Z namespaces/openshift-lws-operator/pods/lws-controller-manager-5c847d9c7c-9t4vb/manager/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.636036014Z namespaces/openshift-lws-operator/pods/lws-controller-manager-5c847d9c7c-9t4vb/manager/manager/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.636041114Z namespaces/openshift-lws-operator/pods/lws-controller-manager-5c847d9c7c-9t4vb/manager/manager/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.636090216Z namespaces/openshift-lws-operator/pods/lws-controller-manager-5c847d9c7c-9t4vb/manager/manager/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.636415122Z namespaces/openshift-lws-operator/pods/lws-controller-manager-5c847d9c7c-9t4vb/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.636480825Z namespaces/openshift-lws-operator/pods/lws-controller-manager-5c847d9c7c-9t4vb/manager/manager/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.636533687Z namespaces/openshift-lws-operator/pods/lws-controller-manager-5c847d9c7c-s7n7k/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.636569219Z namespaces/openshift-lws-operator/pods/lws-controller-manager-5c847d9c7c-s7n7k/lws-controller-manager-5c847d9c7c-s7n7k.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.636651023Z namespaces/openshift-lws-operator/pods/lws-controller-manager-5c847d9c7c-s7n7k/manager/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.636658323Z namespaces/openshift-lws-operator/pods/lws-controller-manager-5c847d9c7c-s7n7k/manager/manager/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.636662563Z namespaces/openshift-lws-operator/pods/lws-controller-manager-5c847d9c7c-s7n7k/manager/manager/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.636706815Z namespaces/openshift-lws-operator/pods/lws-controller-manager-5c847d9c7c-s7n7k/manager/manager/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.636825061Z namespaces/openshift-lws-operator/pods/lws-controller-manager-5c847d9c7c-s7n7k/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.636894684Z namespaces/openshift-lws-operator/pods/lws-controller-manager-5c847d9c7c-s7n7k/manager/manager/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.636940856Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-rd9zg/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.636980588Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-rd9zg/openshift-lws-operator-fd8ccff4c-rd9zg.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.637085053Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-rd9zg/openshift-lws-operator/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.637096874Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-rd9zg/openshift-lws-operator/openshift-lws-operator/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.637107154Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-rd9zg/openshift-lws-operator/openshift-lws-operator/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.637143976Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-rd9zg/openshift-lws-operator/openshift-lws-operator/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.637306784Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-rd9zg/openshift-lws-operator/openshift-lws-operator/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.637376007Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-rd9zg/openshift-lws-operator/openshift-lws-operator/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.637417519Z namespaces/openshift-lws-operator/policy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.637452111Z namespaces/openshift-lws-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.637504323Z namespaces/openshift-lws-operator/route.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.637544395Z namespaces/openshift-lws-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.637592947Z namespaces/openshift-machine-api/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.637601058Z namespaces/openshift-machine-api/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.637608638Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.63764168Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.637796197Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.637973825Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.638152793Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.638350963Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.638535151Z namespaces/openshift-machine-config-operator/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.638542612Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.638547002Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.638581514Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/prometheusrules/machine-config-controller.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.638686069Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/prometheusrules/machine-config-daemon.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.638759392Z namespaces/openshift-machine-config-operator/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.638768953Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.638806674Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.638957301Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.639234074Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.639395522Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.639592481Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.63978148Z namespaces/openshift-marketplace/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.63979195Z namespaces/openshift-marketplace/monitoring.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.639798001Z namespaces/openshift-marketplace/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.639834932Z namespaces/openshift-marketplace/monitoring.coreos.com/prometheusrules/marketplace-alert-rules.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.639899595Z namespaces/openshift-marketplace/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.639906766Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.640032902Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/certified-operators.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.640123796Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/community-operators.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.64020387Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/redhat-marketplace.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.640286004Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/redhat-operators.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.640359307Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.640404949Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.640665452Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.64084987Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.640982257Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.641192647Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.641432428Z namespaces/openshift-monitoring/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.64147769Z namespaces/openshift-monitoring/openshift-monitoring.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.641554773Z namespaces/openshift-monitoring/apps.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.641605596Z namespaces/openshift-monitoring/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.641668709Z namespaces/openshift-monitoring/apps/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.641712021Z namespaces/openshift-monitoring/apps/daemonsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.641850958Z namespaces/openshift-monitoring/apps/deployments.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.64232541Z namespaces/openshift-monitoring/apps/replicasets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.642684007Z namespaces/openshift-monitoring/apps/statefulsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.642913648Z namespaces/openshift-monitoring/autoscaling/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.642947929Z namespaces/openshift-monitoring/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.642997901Z namespaces/openshift-monitoring/batch/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.643061964Z namespaces/openshift-monitoring/batch/cronjobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.643149028Z namespaces/openshift-monitoring/batch/jobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.643231892Z namespaces/openshift-monitoring/build.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.643265724Z namespaces/openshift-monitoring/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.643368319Z namespaces/openshift-monitoring/build.openshift.io/builds.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.643402971Z namespaces/openshift-monitoring/core/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.643453053Z namespaces/openshift-monitoring/core/configmaps.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.646678334Z namespaces/openshift-monitoring/core/endpoints.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.646825581Z namespaces/openshift-monitoring/core/events.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.647835279Z namespaces/openshift-monitoring/core/persistentvolumeclaims.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.647911233Z namespaces/openshift-monitoring/core/pods.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.648952632Z namespaces/openshift-monitoring/core/replicationcontrollers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.649102069Z namespaces/openshift-monitoring/core/secrets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.649614133Z namespaces/openshift-monitoring/core/services.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.64978094Z namespaces/openshift-monitoring/discovery.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.649807282Z namespaces/openshift-monitoring/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.649971149Z namespaces/openshift-monitoring/image.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.650018852Z namespaces/openshift-monitoring/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.650115016Z namespaces/openshift-monitoring/k8s.ovn.org/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.650229632Z namespaces/openshift-monitoring/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.650312165Z namespaces/openshift-monitoring/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.650394149Z namespaces/openshift-monitoring/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.650435551Z namespaces/openshift-monitoring/monitoring.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.650470103Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.650604729Z namespaces/openshift-monitoring/monitoring.coreos.com/alertmanagers/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.650637921Z namespaces/openshift-monitoring/monitoring.coreos.com/alertmanagers/main.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.650738896Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheuses/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.650781258Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheuses/k8s.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.650888043Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.650922504Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/alertmanager-main-rules.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.6510486Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/cluster-monitoring-operator-prometheus-rules.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.6512604Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/kube-state-metrics-rules.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.651363475Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/kubernetes-monitoring-rules.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.651611627Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/node-exporter-rules.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.651745143Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/prometheus-k8s-prometheus-rules.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.651871569Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/prometheus-k8s-thanos-sidecar-rules.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.651960833Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/prometheus-operator-rules.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.65210656Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/telemetry.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.652200544Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/thanos-querier.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.652269918Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.65231598Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/alertmanager-main.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.652413335Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/cluster-monitoring-operator.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.652493588Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/kube-state-metrics.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.652579822Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/kubelet.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.652708428Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/metrics-server.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.652791962Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/node-exporter.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.652942219Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/openshift-state-metrics.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.653053654Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/prometheus-k8s.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.653146959Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/prometheus-operator.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.653234523Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/telemeter-client.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.653325897Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/thanos-querier.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.653408871Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/thanos-sidecar.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.653483665Z namespaces/openshift-monitoring/networking.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.653526257Z namespaces/openshift-monitoring/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.653582689Z namespaces/openshift-monitoring/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.65359292Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.653622381Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.65381055Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.654071042Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.65424102Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.65444167Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.65465737Z namespaces/openshift-monitoring/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.654709572Z namespaces/openshift-monitoring/operators.coreos.com/operatorgroups/openshift-cluster-monitoring.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.654804997Z namespaces/openshift-monitoring/pods/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.654812357Z namespaces/openshift-monitoring/pods/alertmanager-main-0/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.654888161Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager-main-0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.655161914Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.655174724Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.655179694Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.655226437Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.655345402Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.655416366Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.655445437Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.655451197Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.655454967Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.655499349Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.655603125Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.655669628Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.655703509Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.655711199Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.65571619Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.655758292Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.655851446Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.65592214Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.655955841Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.655963731Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.655968552Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.656027185Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.65613413Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.656206173Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.656233264Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.656238214Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.656245145Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.656288117Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.656381081Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.656451334Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.656488436Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.656495176Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.656499677Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.656542699Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.656631053Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.656698116Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.656738698Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.656752019Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.65678245Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.656837052Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.656921567Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.65699075Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.657042552Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-vb4q2/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.657102505Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-vb4q2/cluster-monitoring-operator-548b489cb8-vb4q2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.657197469Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-vb4q2/cluster-monitoring-operator/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.65720559Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-vb4q2/cluster-monitoring-operator/cluster-monitoring-operator/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.65720974Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-vb4q2/cluster-monitoring-operator/cluster-monitoring-operator/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.657253592Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-vb4q2/cluster-monitoring-operator/cluster-monitoring-operator/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.658214547Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-vb4q2/cluster-monitoring-operator/cluster-monitoring-operator/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.658285301Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-vb4q2/cluster-monitoring-operator/cluster-monitoring-operator/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.658335643Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-6ksth/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.658403396Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-6ksth/kube-state-metrics-56d9f944b4-6ksth.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.658538943Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-6ksth/kube-rbac-proxy-main/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.658548663Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-6ksth/kube-rbac-proxy-main/kube-rbac-proxy-main/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.658552893Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-6ksth/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.658597945Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-6ksth/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.65869156Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-6ksth/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.658766443Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-6ksth/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.658796375Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-6ksth/kube-rbac-proxy-self/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.658808085Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-6ksth/kube-rbac-proxy-self/kube-rbac-proxy-self/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.658812236Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-6ksth/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.658847447Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-6ksth/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.658937761Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-6ksth/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.659020515Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-6ksth/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.659076028Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-6ksth/kube-state-metrics/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.659083578Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-6ksth/kube-state-metrics/kube-state-metrics/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.659087578Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-6ksth/kube-state-metrics/kube-state-metrics/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.6591329Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-6ksth/kube-state-metrics/kube-state-metrics/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.659409753Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-6ksth/kube-state-metrics/kube-state-metrics/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.659474667Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-6ksth/kube-state-metrics/kube-state-metrics/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.659511638Z namespaces/openshift-monitoring/pods/metrics-server-7fb54f9c7d-2hmrd/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.659557031Z namespaces/openshift-monitoring/pods/metrics-server-7fb54f9c7d-2hmrd/metrics-server-7fb54f9c7d-2hmrd.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.659652575Z namespaces/openshift-monitoring/pods/metrics-server-7fb54f9c7d-2hmrd/metrics-server/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.659667786Z namespaces/openshift-monitoring/pods/metrics-server-7fb54f9c7d-2hmrd/metrics-server/metrics-server/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.659671906Z namespaces/openshift-monitoring/pods/metrics-server-7fb54f9c7d-2hmrd/metrics-server/metrics-server/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.659706067Z namespaces/openshift-monitoring/pods/metrics-server-7fb54f9c7d-2hmrd/metrics-server/metrics-server/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.659797532Z namespaces/openshift-monitoring/pods/metrics-server-7fb54f9c7d-2hmrd/metrics-server/metrics-server/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.659866955Z namespaces/openshift-monitoring/pods/metrics-server-7fb54f9c7d-2hmrd/metrics-server/metrics-server/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.659895746Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-d88rp/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.660016932Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-d88rp/monitoring-plugin-68b7fc9574-d88rp.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.660123587Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-d88rp/monitoring-plugin/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.660131018Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-d88rp/monitoring-plugin/monitoring-plugin/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.660136378Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-d88rp/monitoring-plugin/monitoring-plugin/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.66017338Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-d88rp/monitoring-plugin/monitoring-plugin/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.660274254Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-d88rp/monitoring-plugin/monitoring-plugin/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.66039032Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-d88rp/monitoring-plugin/monitoring-plugin/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.660422051Z namespaces/openshift-monitoring/pods/node-exporter-d7tx4/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.660474584Z namespaces/openshift-monitoring/pods/node-exporter-d7tx4/node-exporter-d7tx4.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.660584119Z namespaces/openshift-monitoring/pods/node-exporter-d7tx4/init-textfile/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.660592669Z namespaces/openshift-monitoring/pods/node-exporter-d7tx4/init-textfile/init-textfile/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.660596679Z namespaces/openshift-monitoring/pods/node-exporter-d7tx4/init-textfile/init-textfile/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.660633301Z namespaces/openshift-monitoring/pods/node-exporter-d7tx4/init-textfile/init-textfile/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.660733086Z namespaces/openshift-monitoring/pods/node-exporter-d7tx4/init-textfile/init-textfile/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.660802919Z namespaces/openshift-monitoring/pods/node-exporter-d7tx4/init-textfile/init-textfile/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.660847361Z namespaces/openshift-monitoring/pods/node-exporter-d7tx4/kube-rbac-proxy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.660855361Z namespaces/openshift-monitoring/pods/node-exporter-d7tx4/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.660861722Z namespaces/openshift-monitoring/pods/node-exporter-d7tx4/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.660899434Z namespaces/openshift-monitoring/pods/node-exporter-d7tx4/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.661017329Z namespaces/openshift-monitoring/pods/node-exporter-d7tx4/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.661171967Z namespaces/openshift-monitoring/pods/node-exporter-d7tx4/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.661221649Z namespaces/openshift-monitoring/pods/node-exporter-d7tx4/node-exporter/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.661229769Z namespaces/openshift-monitoring/pods/node-exporter-d7tx4/node-exporter/node-exporter/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.661236159Z namespaces/openshift-monitoring/pods/node-exporter-d7tx4/node-exporter/node-exporter/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.661282052Z namespaces/openshift-monitoring/pods/node-exporter-d7tx4/node-exporter/node-exporter/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.661397767Z namespaces/openshift-monitoring/pods/node-exporter-d7tx4/node-exporter/node-exporter/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.66146288Z namespaces/openshift-monitoring/pods/node-exporter-d7tx4/node-exporter/node-exporter/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.661521243Z namespaces/openshift-monitoring/pods/node-exporter-rfbrk/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.661550864Z namespaces/openshift-monitoring/pods/node-exporter-rfbrk/node-exporter-rfbrk.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.661661519Z namespaces/openshift-monitoring/pods/node-exporter-rfbrk/init-textfile/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.66166879Z namespaces/openshift-monitoring/pods/node-exporter-rfbrk/init-textfile/init-textfile/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.6616739Z namespaces/openshift-monitoring/pods/node-exporter-rfbrk/init-textfile/init-textfile/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.661712442Z namespaces/openshift-monitoring/pods/node-exporter-rfbrk/init-textfile/init-textfile/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.661783525Z namespaces/openshift-monitoring/pods/node-exporter-rfbrk/init-textfile/init-textfile/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.661849268Z namespaces/openshift-monitoring/pods/node-exporter-rfbrk/init-textfile/init-textfile/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.66187816Z namespaces/openshift-monitoring/pods/node-exporter-rfbrk/kube-rbac-proxy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.66188459Z namespaces/openshift-monitoring/pods/node-exporter-rfbrk/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.6618886Z namespaces/openshift-monitoring/pods/node-exporter-rfbrk/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.661921652Z namespaces/openshift-monitoring/pods/node-exporter-rfbrk/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.662017066Z namespaces/openshift-monitoring/pods/node-exporter-rfbrk/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.662111711Z namespaces/openshift-monitoring/pods/node-exporter-rfbrk/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.662138982Z namespaces/openshift-monitoring/pods/node-exporter-rfbrk/node-exporter/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.662143872Z namespaces/openshift-monitoring/pods/node-exporter-rfbrk/node-exporter/node-exporter/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.662150053Z namespaces/openshift-monitoring/pods/node-exporter-rfbrk/node-exporter/node-exporter/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.662201195Z namespaces/openshift-monitoring/pods/node-exporter-rfbrk/node-exporter/node-exporter/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.66230654Z namespaces/openshift-monitoring/pods/node-exporter-rfbrk/node-exporter/node-exporter/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.662374753Z namespaces/openshift-monitoring/pods/node-exporter-rfbrk/node-exporter/node-exporter/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.662406805Z namespaces/openshift-monitoring/pods/node-exporter-tpnck/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.662438976Z namespaces/openshift-monitoring/pods/node-exporter-tpnck/node-exporter-tpnck.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.662540831Z namespaces/openshift-monitoring/pods/node-exporter-tpnck/init-textfile/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.662549011Z namespaces/openshift-monitoring/pods/node-exporter-tpnck/init-textfile/init-textfile/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.662552901Z namespaces/openshift-monitoring/pods/node-exporter-tpnck/init-textfile/init-textfile/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.662594943Z namespaces/openshift-monitoring/pods/node-exporter-tpnck/init-textfile/init-textfile/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.662669267Z namespaces/openshift-monitoring/pods/node-exporter-tpnck/init-textfile/init-textfile/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.66273744Z namespaces/openshift-monitoring/pods/node-exporter-tpnck/init-textfile/init-textfile/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.662768282Z namespaces/openshift-monitoring/pods/node-exporter-tpnck/kube-rbac-proxy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.662776522Z namespaces/openshift-monitoring/pods/node-exporter-tpnck/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.662780612Z namespaces/openshift-monitoring/pods/node-exporter-tpnck/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.662820904Z namespaces/openshift-monitoring/pods/node-exporter-tpnck/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.662911068Z namespaces/openshift-monitoring/pods/node-exporter-tpnck/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.662975712Z namespaces/openshift-monitoring/pods/node-exporter-tpnck/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.663032574Z namespaces/openshift-monitoring/pods/node-exporter-tpnck/node-exporter/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.663045895Z namespaces/openshift-monitoring/pods/node-exporter-tpnck/node-exporter/node-exporter/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.663052835Z namespaces/openshift-monitoring/pods/node-exporter-tpnck/node-exporter/node-exporter/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.663099197Z namespaces/openshift-monitoring/pods/node-exporter-tpnck/node-exporter/node-exporter/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.663208902Z namespaces/openshift-monitoring/pods/node-exporter-tpnck/node-exporter/node-exporter/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.663280436Z namespaces/openshift-monitoring/pods/node-exporter-tpnck/node-exporter/node-exporter/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.663319097Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-rsxwn/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.663358399Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-rsxwn/openshift-state-metrics-b8f49f5c6-rsxwn.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.663493466Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-rsxwn/kube-rbac-proxy-main/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.663501966Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-rsxwn/kube-rbac-proxy-main/kube-rbac-proxy-main/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.663506166Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-rsxwn/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.663544538Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-rsxwn/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.663639152Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-rsxwn/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.663705036Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-rsxwn/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.663743677Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-rsxwn/kube-rbac-proxy-self/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.663751048Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-rsxwn/kube-rbac-proxy-self/kube-rbac-proxy-self/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.663760058Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-rsxwn/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.66380527Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-rsxwn/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.663894055Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-rsxwn/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.663961898Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-rsxwn/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.663999789Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-rsxwn/openshift-state-metrics/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.664022981Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-rsxwn/openshift-state-metrics/openshift-state-metrics/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.664055792Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-rsxwn/openshift-state-metrics/openshift-state-metrics/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.664104925Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-rsxwn/openshift-state-metrics/openshift-state-metrics/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.664208609Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-rsxwn/openshift-state-metrics/openshift-state-metrics/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.664276753Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-rsxwn/openshift-state-metrics/openshift-state-metrics/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.664345086Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.664409059Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus-k8s-0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.664588347Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.664595657Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.664604458Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.66464742Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.664749925Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.664815518Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.664847119Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.66485419Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.66486378Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.664915243Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.665020457Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.665105832Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.665132953Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.665141053Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.665144883Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.665184835Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.66527466Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.665344173Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.665378734Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.665385335Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.665392525Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.665424436Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.665517271Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.665585754Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.665626386Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.665634817Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.665638837Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.665668298Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.665757142Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.665823315Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.665853647Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.665860087Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.665869568Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.665906669Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.666634884Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.666703317Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.666748509Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.666757139Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.666761299Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.666799181Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.666896996Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.666964199Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.666994861Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-flfln/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.667072104Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-flfln/prometheus-operator-84d69f794d-flfln.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.66719726Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-flfln/kube-rbac-proxy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.66720581Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-flfln/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.667214951Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-flfln/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.667247642Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-flfln/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.667357267Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-flfln/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.667425021Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-flfln/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.667449852Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-flfln/prometheus-operator/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.667454562Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-flfln/prometheus-operator/prometheus-operator/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.667464012Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-flfln/prometheus-operator/prometheus-operator/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.667512275Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-flfln/prometheus-operator/prometheus-operator/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.667999248Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-flfln/prometheus-operator/prometheus-operator/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.668151345Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-flfln/prometheus-operator/prometheus-operator/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.668199167Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-2dqsh/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.668241929Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-2dqsh/prometheus-operator-admission-webhook-7f8d78f54b-2dqsh.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.668367295Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-2dqsh/prometheus-operator-admission-webhook/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.668374635Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-2dqsh/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.668379035Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-2dqsh/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.668426408Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-2dqsh/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.668520642Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-2dqsh/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.668589765Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-2dqsh/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.668644588Z namespaces/openshift-monitoring/pods/telemeter-client-5f995bbb7b-q4trf/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.668717511Z namespaces/openshift-monitoring/pods/telemeter-client-5f995bbb7b-q4trf/telemeter-client-5f995bbb7b-q4trf.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.668862508Z namespaces/openshift-monitoring/pods/telemeter-client-5f995bbb7b-q4trf/kube-rbac-proxy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.668870429Z namespaces/openshift-monitoring/pods/telemeter-client-5f995bbb7b-q4trf/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.668874329Z namespaces/openshift-monitoring/pods/telemeter-client-5f995bbb7b-q4trf/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.668915471Z namespaces/openshift-monitoring/pods/telemeter-client-5f995bbb7b-q4trf/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.669021626Z namespaces/openshift-monitoring/pods/telemeter-client-5f995bbb7b-q4trf/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.66910592Z namespaces/openshift-monitoring/pods/telemeter-client-5f995bbb7b-q4trf/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.669134011Z namespaces/openshift-monitoring/pods/telemeter-client-5f995bbb7b-q4trf/reload/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.669138811Z namespaces/openshift-monitoring/pods/telemeter-client-5f995bbb7b-q4trf/reload/reload/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.669144991Z namespaces/openshift-monitoring/pods/telemeter-client-5f995bbb7b-q4trf/reload/reload/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.669192704Z namespaces/openshift-monitoring/pods/telemeter-client-5f995bbb7b-q4trf/reload/reload/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.669312619Z namespaces/openshift-monitoring/pods/telemeter-client-5f995bbb7b-q4trf/reload/reload/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.669380943Z namespaces/openshift-monitoring/pods/telemeter-client-5f995bbb7b-q4trf/reload/reload/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.669411964Z namespaces/openshift-monitoring/pods/telemeter-client-5f995bbb7b-q4trf/telemeter-client/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.669420544Z namespaces/openshift-monitoring/pods/telemeter-client-5f995bbb7b-q4trf/telemeter-client/telemeter-client/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.669424505Z namespaces/openshift-monitoring/pods/telemeter-client-5f995bbb7b-q4trf/telemeter-client/telemeter-client/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.669483198Z namespaces/openshift-monitoring/pods/telemeter-client-5f995bbb7b-q4trf/telemeter-client/telemeter-client/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.669576342Z namespaces/openshift-monitoring/pods/telemeter-client-5f995bbb7b-q4trf/telemeter-client/telemeter-client/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.669642895Z namespaces/openshift-monitoring/pods/telemeter-client-5f995bbb7b-q4trf/telemeter-client/telemeter-client/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.669670646Z namespaces/openshift-monitoring/pods/thanos-querier-bbf8fcf7d-wk55s/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.66974281Z namespaces/openshift-monitoring/pods/thanos-querier-bbf8fcf7d-wk55s/thanos-querier-bbf8fcf7d-wk55s.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.669873296Z namespaces/openshift-monitoring/pods/thanos-querier-bbf8fcf7d-wk55s/kube-rbac-proxy-metrics/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.669880516Z namespaces/openshift-monitoring/pods/thanos-querier-bbf8fcf7d-wk55s/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.669884446Z namespaces/openshift-monitoring/pods/thanos-querier-bbf8fcf7d-wk55s/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.669920098Z namespaces/openshift-monitoring/pods/thanos-querier-bbf8fcf7d-wk55s/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.670026853Z namespaces/openshift-monitoring/pods/thanos-querier-bbf8fcf7d-wk55s/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.670100907Z namespaces/openshift-monitoring/pods/thanos-querier-bbf8fcf7d-wk55s/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.670143678Z namespaces/openshift-monitoring/pods/thanos-querier-bbf8fcf7d-wk55s/kube-rbac-proxy-rules/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.670151679Z namespaces/openshift-monitoring/pods/thanos-querier-bbf8fcf7d-wk55s/kube-rbac-proxy-rules/kube-rbac-proxy-rules/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.670159199Z namespaces/openshift-monitoring/pods/thanos-querier-bbf8fcf7d-wk55s/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.670192331Z namespaces/openshift-monitoring/pods/thanos-querier-bbf8fcf7d-wk55s/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.670284715Z namespaces/openshift-monitoring/pods/thanos-querier-bbf8fcf7d-wk55s/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.670354339Z namespaces/openshift-monitoring/pods/thanos-querier-bbf8fcf7d-wk55s/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.67039365Z namespaces/openshift-monitoring/pods/thanos-querier-bbf8fcf7d-wk55s/kube-rbac-proxy-web/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.670401111Z namespaces/openshift-monitoring/pods/thanos-querier-bbf8fcf7d-wk55s/kube-rbac-proxy-web/kube-rbac-proxy-web/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.670405201Z namespaces/openshift-monitoring/pods/thanos-querier-bbf8fcf7d-wk55s/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.670435872Z namespaces/openshift-monitoring/pods/thanos-querier-bbf8fcf7d-wk55s/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.670538787Z namespaces/openshift-monitoring/pods/thanos-querier-bbf8fcf7d-wk55s/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.67060566Z namespaces/openshift-monitoring/pods/thanos-querier-bbf8fcf7d-wk55s/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.670620061Z namespaces/openshift-monitoring/pods/thanos-querier-bbf8fcf7d-wk55s/kube-rbac-proxy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.670629081Z namespaces/openshift-monitoring/pods/thanos-querier-bbf8fcf7d-wk55s/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.670646822Z namespaces/openshift-monitoring/pods/thanos-querier-bbf8fcf7d-wk55s/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.670707875Z namespaces/openshift-monitoring/pods/thanos-querier-bbf8fcf7d-wk55s/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.67080192Z namespaces/openshift-monitoring/pods/thanos-querier-bbf8fcf7d-wk55s/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.670867183Z namespaces/openshift-monitoring/pods/thanos-querier-bbf8fcf7d-wk55s/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.670881923Z namespaces/openshift-monitoring/pods/thanos-querier-bbf8fcf7d-wk55s/prom-label-proxy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.670894604Z namespaces/openshift-monitoring/pods/thanos-querier-bbf8fcf7d-wk55s/prom-label-proxy/prom-label-proxy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.670900924Z namespaces/openshift-monitoring/pods/thanos-querier-bbf8fcf7d-wk55s/prom-label-proxy/prom-label-proxy/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.670951067Z namespaces/openshift-monitoring/pods/thanos-querier-bbf8fcf7d-wk55s/prom-label-proxy/prom-label-proxy/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.671064502Z namespaces/openshift-monitoring/pods/thanos-querier-bbf8fcf7d-wk55s/prom-label-proxy/prom-label-proxy/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.671159617Z namespaces/openshift-monitoring/pods/thanos-querier-bbf8fcf7d-wk55s/prom-label-proxy/prom-label-proxy/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.671175297Z namespaces/openshift-monitoring/pods/thanos-querier-bbf8fcf7d-wk55s/thanos-query/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.671183458Z namespaces/openshift-monitoring/pods/thanos-querier-bbf8fcf7d-wk55s/thanos-query/thanos-query/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.671188378Z namespaces/openshift-monitoring/pods/thanos-querier-bbf8fcf7d-wk55s/thanos-query/thanos-query/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.67123935Z namespaces/openshift-monitoring/pods/thanos-querier-bbf8fcf7d-wk55s/thanos-query/thanos-query/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.671342215Z namespaces/openshift-monitoring/pods/thanos-querier-bbf8fcf7d-wk55s/thanos-query/thanos-query/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.671416829Z namespaces/openshift-monitoring/pods/thanos-querier-bbf8fcf7d-wk55s/thanos-query/thanos-query/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.671461551Z namespaces/openshift-monitoring/policy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.671477001Z namespaces/openshift-monitoring/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.671538224Z namespaces/openshift-monitoring/route.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.671567996Z namespaces/openshift-monitoring/route.openshift.io/routes.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.671691742Z namespaces/openshift-multus/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.671738554Z namespaces/openshift-multus/openshift-multus.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.671849399Z namespaces/openshift-multus/apps.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.671889421Z namespaces/openshift-multus/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.671924973Z namespaces/openshift-multus/apps/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.671979155Z namespaces/openshift-multus/apps/daemonsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.672229217Z namespaces/openshift-multus/apps/deployments.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.67230894Z namespaces/openshift-multus/apps/replicasets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.672385904Z namespaces/openshift-multus/apps/statefulsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.672433416Z namespaces/openshift-multus/apps/daemonsets/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.672474798Z namespaces/openshift-multus/apps/daemonsets/multus-additional-cni-plugins.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.672618075Z namespaces/openshift-multus/apps/daemonsets/multus.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.672762812Z namespaces/openshift-multus/apps/daemonsets/network-metrics-daemon.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.672834965Z namespaces/openshift-multus/autoscaling/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.672919429Z namespaces/openshift-multus/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.672964881Z namespaces/openshift-multus/batch/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.673022064Z namespaces/openshift-multus/batch/cronjobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.673114528Z namespaces/openshift-multus/batch/jobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.6731575Z namespaces/openshift-multus/build.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.673191562Z namespaces/openshift-multus/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.673272726Z namespaces/openshift-multus/build.openshift.io/builds.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.673349079Z namespaces/openshift-multus/core/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.673384801Z namespaces/openshift-multus/core/configmaps.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.673532048Z namespaces/openshift-multus/core/endpoints.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.673616512Z namespaces/openshift-multus/core/events.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.674126466Z namespaces/openshift-multus/core/persistentvolumeclaims.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.674245491Z namespaces/openshift-multus/core/pods.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.674913003Z namespaces/openshift-multus/core/replicationcontrollers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.675025498Z namespaces/openshift-multus/core/secrets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.675263659Z namespaces/openshift-multus/core/services.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.675344513Z namespaces/openshift-multus/core/configmaps/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.675393005Z namespaces/openshift-multus/core/configmaps/cni-copy-resources.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.67549398Z namespaces/openshift-multus/core/configmaps/default-cni-sysctl-allowlist.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.675577554Z namespaces/openshift-multus/core/configmaps/multus-daemon-config.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.675665288Z namespaces/openshift-multus/core/configmaps/whereabouts-flatfile-config.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.675757563Z namespaces/openshift-multus/core/serviceaccounts/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.675799835Z namespaces/openshift-multus/core/serviceaccounts/metrics-daemon-sa.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.675888809Z namespaces/openshift-multus/core/serviceaccounts/multus-ac.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.675968423Z namespaces/openshift-multus/core/serviceaccounts/multus-ancillary-tools.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.676074138Z namespaces/openshift-multus/core/serviceaccounts/multus.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.67612457Z namespaces/openshift-multus/core/services/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.676163672Z namespaces/openshift-multus/core/services/network-metrics-service.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.676220595Z namespaces/openshift-multus/discovery.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.676264477Z namespaces/openshift-multus/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.676319419Z namespaces/openshift-multus/image.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.676363821Z namespaces/openshift-multus/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.676410984Z namespaces/openshift-multus/k8s.ovn.org/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.676453135Z namespaces/openshift-multus/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.676536259Z namespaces/openshift-multus/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.676616493Z namespaces/openshift-multus/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.676669586Z namespaces/openshift-multus/monitoring.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.676713698Z namespaces/openshift-multus/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.67676962Z namespaces/openshift-multus/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.676803742Z namespaces/openshift-multus/monitoring.coreos.com/servicemonitors/monitor-network.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.676860265Z namespaces/openshift-multus/networking.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.676903717Z namespaces/openshift-multus/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.676948129Z namespaces/openshift-multus/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.67696155Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.677027293Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.677214621Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.67739792Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.677536926Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.677734126Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.678121954Z namespaces/openshift-multus/pods/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.678134134Z namespaces/openshift-multus/pods/multus-6nndf/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.678162266Z namespaces/openshift-multus/pods/multus-6nndf/multus-6nndf.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.678268201Z namespaces/openshift-multus/pods/multus-6nndf/kube-multus/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.678278511Z namespaces/openshift-multus/pods/multus-6nndf/kube-multus/kube-multus/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.678284332Z namespaces/openshift-multus/pods/multus-6nndf/kube-multus/kube-multus/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.678322243Z namespaces/openshift-multus/pods/multus-6nndf/kube-multus/kube-multus/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.679034277Z namespaces/openshift-multus/pods/multus-6nndf/kube-multus/kube-multus/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.679124801Z namespaces/openshift-multus/pods/multus-6nndf/kube-multus/kube-multus/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.679165133Z namespaces/openshift-multus/pods/multus-8lmdp/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.679205025Z namespaces/openshift-multus/pods/multus-8lmdp/multus-8lmdp.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.679303809Z namespaces/openshift-multus/pods/multus-8lmdp/kube-multus/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.67931703Z namespaces/openshift-multus/pods/multus-8lmdp/kube-multus/kube-multus/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.679324361Z namespaces/openshift-multus/pods/multus-8lmdp/kube-multus/kube-multus/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.679357742Z namespaces/openshift-multus/pods/multus-8lmdp/kube-multus/kube-multus/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.67994842Z namespaces/openshift-multus/pods/multus-8lmdp/kube-multus/kube-multus/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.680021503Z namespaces/openshift-multus/pods/multus-8lmdp/kube-multus/kube-multus/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.680177321Z namespaces/openshift-multus/pods/multus-9ncwb/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.680205122Z namespaces/openshift-multus/pods/multus-9ncwb/multus-9ncwb.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.680305957Z namespaces/openshift-multus/pods/multus-9ncwb/kube-multus/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.680313297Z namespaces/openshift-multus/pods/multus-9ncwb/kube-multus/kube-multus/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.680318007Z namespaces/openshift-multus/pods/multus-9ncwb/kube-multus/kube-multus/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.680348759Z namespaces/openshift-multus/pods/multus-9ncwb/kube-multus/kube-multus/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.681189258Z namespaces/openshift-multus/pods/multus-9ncwb/kube-multus/kube-multus/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.681290713Z namespaces/openshift-multus/pods/multus-9ncwb/kube-multus/kube-multus/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.681359566Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-45d2k/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.681408329Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-45d2k/multus-additional-cni-plugins-45d2k.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.681561596Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-45d2k/bond-cni-plugin/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.681570106Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-45d2k/bond-cni-plugin/bond-cni-plugin/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.681575116Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-45d2k/bond-cni-plugin/bond-cni-plugin/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.681598828Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-45d2k/bond-cni-plugin/bond-cni-plugin/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.681695622Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-45d2k/bond-cni-plugin/bond-cni-plugin/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.681762895Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-45d2k/bond-cni-plugin/bond-cni-plugin/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.681803497Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-45d2k/cni-plugins/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.681811038Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-45d2k/cni-plugins/cni-plugins/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.681815088Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-45d2k/cni-plugins/cni-plugins/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.681851709Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-45d2k/cni-plugins/cni-plugins/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.681952514Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-45d2k/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.682020968Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-45d2k/cni-plugins/cni-plugins/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.682087271Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-45d2k/egress-router-binary-copy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.682094671Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-45d2k/egress-router-binary-copy/egress-router-binary-copy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.682098871Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-45d2k/egress-router-binary-copy/egress-router-binary-copy/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.682136673Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-45d2k/egress-router-binary-copy/egress-router-binary-copy/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.682233078Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-45d2k/egress-router-binary-copy/egress-router-binary-copy/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.682304051Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-45d2k/egress-router-binary-copy/egress-router-binary-copy/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.682339632Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-45d2k/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.682345973Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-45d2k/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.682349933Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-45d2k/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.682391305Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-45d2k/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.682458718Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-45d2k/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.682537822Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-45d2k/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.682565983Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-45d2k/routeoverride-cni/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.682571233Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-45d2k/routeoverride-cni/routeoverride-cni/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.682575063Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-45d2k/routeoverride-cni/routeoverride-cni/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.682617965Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-45d2k/routeoverride-cni/routeoverride-cni/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.68270696Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-45d2k/routeoverride-cni/routeoverride-cni/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.682780373Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-45d2k/routeoverride-cni/routeoverride-cni/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.682817845Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-45d2k/whereabouts-cni-bincopy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.682825205Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-45d2k/whereabouts-cni-bincopy/whereabouts-cni-bincopy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.682829175Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-45d2k/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.682854076Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-45d2k/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.682981382Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-45d2k/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.683075317Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-45d2k/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.683118999Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-45d2k/whereabouts-cni/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.683129249Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-45d2k/whereabouts-cni/whereabouts-cni/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.68313561Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-45d2k/whereabouts-cni/whereabouts-cni/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.68314473Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-45d2k/whereabouts-cni/whereabouts-cni/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.683242485Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-45d2k/whereabouts-cni/whereabouts-cni/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.68334478Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-45d2k/whereabouts-cni/whereabouts-cni/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.683363551Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lk64z/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.683414323Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lk64z/multus-additional-cni-plugins-lk64z.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.683544029Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lk64z/bond-cni-plugin/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.68355444Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lk64z/bond-cni-plugin/bond-cni-plugin/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.68356038Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lk64z/bond-cni-plugin/bond-cni-plugin/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.683614892Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lk64z/bond-cni-plugin/bond-cni-plugin/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.683713667Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lk64z/bond-cni-plugin/bond-cni-plugin/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.68378549Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lk64z/bond-cni-plugin/bond-cni-plugin/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.683824982Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lk64z/cni-plugins/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.683832773Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lk64z/cni-plugins/cni-plugins/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.683843693Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lk64z/cni-plugins/cni-plugins/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.683872695Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lk64z/cni-plugins/cni-plugins/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.683959939Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lk64z/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.684061904Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lk64z/cni-plugins/cni-plugins/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.684093505Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lk64z/egress-router-binary-copy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.684098465Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lk64z/egress-router-binary-copy/egress-router-binary-copy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.684102415Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lk64z/egress-router-binary-copy/egress-router-binary-copy/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.684161328Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lk64z/egress-router-binary-copy/egress-router-binary-copy/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.684246672Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lk64z/egress-router-binary-copy/egress-router-binary-copy/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.684319396Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lk64z/egress-router-binary-copy/egress-router-binary-copy/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.684352597Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lk64z/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.684362867Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lk64z/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.684372108Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lk64z/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.68440579Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lk64z/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.684467812Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lk64z/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.684546056Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lk64z/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.684579668Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lk64z/routeoverride-cni/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.684586238Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lk64z/routeoverride-cni/routeoverride-cni/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.684590278Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lk64z/routeoverride-cni/routeoverride-cni/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.68462613Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lk64z/routeoverride-cni/routeoverride-cni/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.684716644Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lk64z/routeoverride-cni/routeoverride-cni/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.684790688Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lk64z/routeoverride-cni/routeoverride-cni/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.684825129Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lk64z/whereabouts-cni-bincopy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.68483371Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lk64z/whereabouts-cni-bincopy/whereabouts-cni-bincopy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.684867101Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lk64z/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.684901533Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lk64z/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.684996508Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lk64z/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.685084212Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lk64z/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.685118493Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lk64z/whereabouts-cni/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.685128624Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lk64z/whereabouts-cni/whereabouts-cni/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.685136014Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lk64z/whereabouts-cni/whereabouts-cni/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.685170646Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lk64z/whereabouts-cni/whereabouts-cni/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.68525204Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lk64z/whereabouts-cni/whereabouts-cni/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.685376225Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lk64z/whereabouts-cni/whereabouts-cni/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.685409547Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-rpbv6/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.685456809Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-rpbv6/multus-additional-cni-plugins-rpbv6.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.685610296Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-rpbv6/bond-cni-plugin/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.685618387Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-rpbv6/bond-cni-plugin/bond-cni-plugin/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.685622637Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-rpbv6/bond-cni-plugin/bond-cni-plugin/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.685660889Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-rpbv6/bond-cni-plugin/bond-cni-plugin/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.685750133Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-rpbv6/bond-cni-plugin/bond-cni-plugin/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.685817266Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-rpbv6/bond-cni-plugin/bond-cni-plugin/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.685850628Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-rpbv6/cni-plugins/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.685859648Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-rpbv6/cni-plugins/cni-plugins/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.685863658Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-rpbv6/cni-plugins/cni-plugins/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.68590229Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-rpbv6/cni-plugins/cni-plugins/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.685985784Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-rpbv6/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.686080868Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-rpbv6/cni-plugins/cni-plugins/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.68612013Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-rpbv6/egress-router-binary-copy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.686126981Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-rpbv6/egress-router-binary-copy/egress-router-binary-copy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.686130851Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-rpbv6/egress-router-binary-copy/egress-router-binary-copy/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.686157842Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-rpbv6/egress-router-binary-copy/egress-router-binary-copy/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.686249366Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-rpbv6/egress-router-binary-copy/egress-router-binary-copy/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.686365022Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-rpbv6/egress-router-binary-copy/egress-router-binary-copy/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.686400403Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-rpbv6/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.686408624Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-rpbv6/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.686412644Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-rpbv6/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.686445725Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-rpbv6/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.686520019Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-rpbv6/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.686593263Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-rpbv6/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.686616824Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-rpbv6/routeoverride-cni/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.686623544Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-rpbv6/routeoverride-cni/routeoverride-cni/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.686627424Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-rpbv6/routeoverride-cni/routeoverride-cni/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.686673426Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-rpbv6/routeoverride-cni/routeoverride-cni/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.686778061Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-rpbv6/routeoverride-cni/routeoverride-cni/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.686844344Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-rpbv6/routeoverride-cni/routeoverride-cni/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.686863325Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-rpbv6/whereabouts-cni-bincopy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.686875526Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-rpbv6/whereabouts-cni-bincopy/whereabouts-cni-bincopy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.686880616Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-rpbv6/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.686925988Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-rpbv6/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.687014852Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-rpbv6/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.687103916Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-rpbv6/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.687127658Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-rpbv6/whereabouts-cni/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.687134188Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-rpbv6/whereabouts-cni/whereabouts-cni/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.687144978Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-rpbv6/whereabouts-cni/whereabouts-cni/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.68718686Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-rpbv6/whereabouts-cni/whereabouts-cni/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.687271164Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-rpbv6/whereabouts-cni/whereabouts-cni/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.687353858Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-rpbv6/whereabouts-cni/whereabouts-cni/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.687400021Z namespaces/openshift-multus/pods/network-metrics-daemon-4wb9p/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.687442612Z namespaces/openshift-multus/pods/network-metrics-daemon-4wb9p/network-metrics-daemon-4wb9p.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.687529787Z namespaces/openshift-multus/pods/network-metrics-daemon-4wb9p/kube-rbac-proxy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.687537467Z namespaces/openshift-multus/pods/network-metrics-daemon-4wb9p/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.687541367Z namespaces/openshift-multus/pods/network-metrics-daemon-4wb9p/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.687581029Z namespaces/openshift-multus/pods/network-metrics-daemon-4wb9p/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.687695894Z namespaces/openshift-multus/pods/network-metrics-daemon-4wb9p/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.687764407Z namespaces/openshift-multus/pods/network-metrics-daemon-4wb9p/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.687794619Z namespaces/openshift-multus/pods/network-metrics-daemon-4wb9p/network-metrics-daemon/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.687801089Z namespaces/openshift-multus/pods/network-metrics-daemon-4wb9p/network-metrics-daemon/network-metrics-daemon/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.68781109Z namespaces/openshift-multus/pods/network-metrics-daemon-4wb9p/network-metrics-daemon/network-metrics-daemon/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.687854332Z namespaces/openshift-multus/pods/network-metrics-daemon-4wb9p/network-metrics-daemon/network-metrics-daemon/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.68803196Z namespaces/openshift-multus/pods/network-metrics-daemon-4wb9p/network-metrics-daemon/network-metrics-daemon/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.688108614Z namespaces/openshift-multus/pods/network-metrics-daemon-4wb9p/network-metrics-daemon/network-metrics-daemon/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.688149536Z namespaces/openshift-multus/pods/network-metrics-daemon-5fm96/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.688182517Z namespaces/openshift-multus/pods/network-metrics-daemon-5fm96/network-metrics-daemon-5fm96.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.688261471Z namespaces/openshift-multus/pods/network-metrics-daemon-5fm96/kube-rbac-proxy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.688272951Z namespaces/openshift-multus/pods/network-metrics-daemon-5fm96/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.688278632Z namespaces/openshift-multus/pods/network-metrics-daemon-5fm96/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.688303483Z namespaces/openshift-multus/pods/network-metrics-daemon-5fm96/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.688395037Z namespaces/openshift-multus/pods/network-metrics-daemon-5fm96/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.688462151Z namespaces/openshift-multus/pods/network-metrics-daemon-5fm96/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.688481191Z namespaces/openshift-multus/pods/network-metrics-daemon-5fm96/network-metrics-daemon/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.688489302Z namespaces/openshift-multus/pods/network-metrics-daemon-5fm96/network-metrics-daemon/network-metrics-daemon/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.688499922Z namespaces/openshift-multus/pods/network-metrics-daemon-5fm96/network-metrics-daemon/network-metrics-daemon/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.688554025Z namespaces/openshift-multus/pods/network-metrics-daemon-5fm96/network-metrics-daemon/network-metrics-daemon/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.688733243Z namespaces/openshift-multus/pods/network-metrics-daemon-5fm96/network-metrics-daemon/network-metrics-daemon/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.688833358Z namespaces/openshift-multus/pods/network-metrics-daemon-5fm96/network-metrics-daemon/network-metrics-daemon/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.688856139Z namespaces/openshift-multus/pods/network-metrics-daemon-ddfml/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.688899961Z namespaces/openshift-multus/pods/network-metrics-daemon-ddfml/network-metrics-daemon-ddfml.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.688982745Z namespaces/openshift-multus/pods/network-metrics-daemon-ddfml/kube-rbac-proxy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.688990625Z namespaces/openshift-multus/pods/network-metrics-daemon-ddfml/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.688994536Z namespaces/openshift-multus/pods/network-metrics-daemon-ddfml/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.689141592Z namespaces/openshift-multus/pods/network-metrics-daemon-ddfml/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.689232437Z namespaces/openshift-multus/pods/network-metrics-daemon-ddfml/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.68930277Z namespaces/openshift-multus/pods/network-metrics-daemon-ddfml/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.689328751Z namespaces/openshift-multus/pods/network-metrics-daemon-ddfml/network-metrics-daemon/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.689336032Z namespaces/openshift-multus/pods/network-metrics-daemon-ddfml/network-metrics-daemon/network-metrics-daemon/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.689342182Z namespaces/openshift-multus/pods/network-metrics-daemon-ddfml/network-metrics-daemon/network-metrics-daemon/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.689390944Z namespaces/openshift-multus/pods/network-metrics-daemon-ddfml/network-metrics-daemon/network-metrics-daemon/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.689557732Z namespaces/openshift-multus/pods/network-metrics-daemon-ddfml/network-metrics-daemon/network-metrics-daemon/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.689625975Z namespaces/openshift-multus/pods/network-metrics-daemon-ddfml/network-metrics-daemon/network-metrics-daemon/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.689658187Z namespaces/openshift-multus/policy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.689701029Z namespaces/openshift-multus/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.689751841Z namespaces/openshift-multus/rbac.authorization.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.689759212Z namespaces/openshift-multus/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.689795903Z namespaces/openshift-multus/rbac.authorization.k8s.io/rolebindings/multus-whereabouts.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.689891858Z namespaces/openshift-multus/rbac.authorization.k8s.io/rolebindings/prometheus-k8s.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.68993324Z namespaces/openshift-multus/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.689979552Z namespaces/openshift-multus/rbac.authorization.k8s.io/roles/prometheus-k8s.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.690090457Z namespaces/openshift-multus/rbac.authorization.k8s.io/roles/whereabouts-cni.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.690139049Z namespaces/openshift-multus/route.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.690181401Z namespaces/openshift-multus/route.openshift.io/routes.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.690227094Z namespaces/openshift-must-gather-jjmjm/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.690234584Z namespaces/openshift-must-gather-jjmjm/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.690238654Z namespaces/openshift-must-gather-jjmjm/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.690285236Z namespaces/openshift-must-gather-jjmjm/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.690464625Z namespaces/openshift-must-gather-jjmjm/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.690653464Z namespaces/openshift-must-gather-jjmjm/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.69079096Z namespaces/openshift-must-gather-jjmjm/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.691066963Z namespaces/openshift-must-gather-jjmjm/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.691334686Z namespaces/openshift-network-console/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.691408839Z namespaces/openshift-network-console/openshift-network-console.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.691538515Z namespaces/openshift-network-console/apps.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.691603698Z namespaces/openshift-network-console/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.691659481Z namespaces/openshift-network-console/apps/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.691839019Z namespaces/openshift-network-console/apps/daemonsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.691925253Z namespaces/openshift-network-console/apps/deployments.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.692075961Z namespaces/openshift-network-console/apps/replicasets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.692202746Z namespaces/openshift-network-console/apps/statefulsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.692253719Z namespaces/openshift-network-console/apps/deployments/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.692293301Z namespaces/openshift-network-console/apps/deployments/networking-console-plugin.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.692362554Z namespaces/openshift-network-console/autoscaling/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.692407336Z namespaces/openshift-network-console/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.692454298Z namespaces/openshift-network-console/batch/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.69250239Z namespaces/openshift-network-console/batch/cronjobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.692583774Z namespaces/openshift-network-console/batch/jobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.692628647Z namespaces/openshift-network-console/build.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.692678939Z namespaces/openshift-network-console/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.692754002Z namespaces/openshift-network-console/build.openshift.io/builds.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.692802335Z namespaces/openshift-network-console/core/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.692839396Z namespaces/openshift-network-console/core/configmaps.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.692948622Z namespaces/openshift-network-console/core/endpoints.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.693058627Z namespaces/openshift-network-console/core/events.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.693165872Z namespaces/openshift-network-console/core/persistentvolumeclaims.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.693248886Z namespaces/openshift-network-console/core/pods.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.693413363Z namespaces/openshift-network-console/core/replicationcontrollers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.693540669Z namespaces/openshift-network-console/core/secrets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.693645054Z namespaces/openshift-network-console/core/services.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.693704007Z namespaces/openshift-network-console/core/configmaps/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.693738899Z namespaces/openshift-network-console/core/configmaps/networking-console-plugin.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.693797802Z namespaces/openshift-network-console/core/services/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.693828093Z namespaces/openshift-network-console/core/services/networking-console-plugin.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.693891916Z namespaces/openshift-network-console/discovery.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.693939468Z namespaces/openshift-network-console/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.69398578Z namespaces/openshift-network-console/image.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.694039873Z namespaces/openshift-network-console/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.694098616Z namespaces/openshift-network-console/k8s.ovn.org/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.694138698Z namespaces/openshift-network-console/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.694284604Z namespaces/openshift-network-console/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.694364118Z namespaces/openshift-network-console/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.6944114Z namespaces/openshift-network-console/monitoring.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.694452152Z namespaces/openshift-network-console/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.694502705Z namespaces/openshift-network-console/networking.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.694547767Z namespaces/openshift-network-console/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.694592649Z namespaces/openshift-network-console/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.694600689Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.694633931Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.694803409Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.694980727Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.695395717Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.695596736Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.695777835Z namespaces/openshift-network-console/pods/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.695786995Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-cdrjg/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.695823877Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-cdrjg/networking-console-plugin-74c9f884b7-cdrjg.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.696142442Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-cdrjg/networking-console-plugin/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.696153972Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-cdrjg/networking-console-plugin/networking-console-plugin/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.696158142Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-cdrjg/networking-console-plugin/networking-console-plugin/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.696193524Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-cdrjg/networking-console-plugin/networking-console-plugin/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.696288019Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-cdrjg/networking-console-plugin/networking-console-plugin/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.696388193Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-cdrjg/networking-console-plugin/networking-console-plugin/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.696424265Z namespaces/openshift-network-console/policy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.696466857Z namespaces/openshift-network-console/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.69651793Z namespaces/openshift-network-console/route.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.696560641Z namespaces/openshift-network-console/route.openshift.io/routes.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.696608164Z namespaces/openshift-network-diagnostics/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.696658296Z namespaces/openshift-network-diagnostics/openshift-network-diagnostics.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.696713209Z namespaces/openshift-network-diagnostics/apps.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.696840205Z namespaces/openshift-network-diagnostics/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.697034264Z namespaces/openshift-network-diagnostics/apps/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.697138859Z namespaces/openshift-network-diagnostics/apps/daemonsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.697252664Z namespaces/openshift-network-diagnostics/apps/deployments.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.69737087Z namespaces/openshift-network-diagnostics/apps/replicasets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.697468334Z namespaces/openshift-network-diagnostics/apps/statefulsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.697518117Z namespaces/openshift-network-diagnostics/apps/daemonsets/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.697547268Z namespaces/openshift-network-diagnostics/apps/daemonsets/network-check-target.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.697621241Z namespaces/openshift-network-diagnostics/apps/deployments/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.697667093Z namespaces/openshift-network-diagnostics/apps/deployments/network-check-source.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.697724456Z namespaces/openshift-network-diagnostics/autoscaling/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.697765448Z namespaces/openshift-network-diagnostics/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.69781271Z namespaces/openshift-network-diagnostics/batch/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.697857933Z namespaces/openshift-network-diagnostics/batch/cronjobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.697934826Z namespaces/openshift-network-diagnostics/batch/jobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.697983208Z namespaces/openshift-network-diagnostics/build.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.698052182Z namespaces/openshift-network-diagnostics/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.698132115Z namespaces/openshift-network-diagnostics/build.openshift.io/builds.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.698181808Z namespaces/openshift-network-diagnostics/core/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.698213299Z namespaces/openshift-network-diagnostics/core/configmaps.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.698323955Z namespaces/openshift-network-diagnostics/core/endpoints.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.698407858Z namespaces/openshift-network-diagnostics/core/events.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.698602437Z namespaces/openshift-network-diagnostics/core/persistentvolumeclaims.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.698679801Z namespaces/openshift-network-diagnostics/core/pods.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.698887291Z namespaces/openshift-network-diagnostics/core/replicationcontrollers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.698972665Z namespaces/openshift-network-diagnostics/core/secrets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.699101291Z namespaces/openshift-network-diagnostics/core/services.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.699160054Z namespaces/openshift-network-diagnostics/core/serviceaccounts/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.699204836Z namespaces/openshift-network-diagnostics/core/serviceaccounts/network-diagnostics.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.699255918Z namespaces/openshift-network-diagnostics/core/services/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.69929724Z namespaces/openshift-network-diagnostics/core/services/network-check-source.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.699399075Z namespaces/openshift-network-diagnostics/core/services/network-check-target.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.699433837Z namespaces/openshift-network-diagnostics/discovery.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.699483609Z namespaces/openshift-network-diagnostics/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.699546582Z namespaces/openshift-network-diagnostics/image.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.699587574Z namespaces/openshift-network-diagnostics/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.699638416Z namespaces/openshift-network-diagnostics/k8s.ovn.org/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.699681588Z namespaces/openshift-network-diagnostics/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.699761962Z namespaces/openshift-network-diagnostics/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.699838426Z namespaces/openshift-network-diagnostics/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.699885488Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.699915599Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.699981573Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.700000993Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/servicemonitors/network-check-source.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.700088907Z namespaces/openshift-network-diagnostics/networking.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.70014959Z namespaces/openshift-network-diagnostics/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.700210733Z namespaces/openshift-network-diagnostics/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.700219354Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.700230094Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.700393172Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.700576761Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.700712177Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.700942787Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.701147257Z namespaces/openshift-network-diagnostics/pods/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.701158838Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-kz24q/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.701242602Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-kz24q/network-check-source-7d88cb55d8-kz24q.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.701550386Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-kz24q/check-endpoints/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.701558947Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-kz24q/check-endpoints/check-endpoints/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.701563557Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-kz24q/check-endpoints/check-endpoints/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.701599358Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-kz24q/check-endpoints/check-endpoints/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.701716514Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-kz24q/check-endpoints/check-endpoints/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.701782767Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-kz24q/check-endpoints/check-endpoints/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.701818199Z namespaces/openshift-network-diagnostics/pods/network-check-target-fmxtb/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.70185146Z namespaces/openshift-network-diagnostics/pods/network-check-target-fmxtb/network-check-target-fmxtb.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.701932374Z namespaces/openshift-network-diagnostics/pods/network-check-target-fmxtb/network-check-target-container/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.701940405Z namespaces/openshift-network-diagnostics/pods/network-check-target-fmxtb/network-check-target-container/network-check-target-container/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.701944345Z namespaces/openshift-network-diagnostics/pods/network-check-target-fmxtb/network-check-target-container/network-check-target-container/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.701979526Z namespaces/openshift-network-diagnostics/pods/network-check-target-fmxtb/network-check-target-container/network-check-target-container/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.702086872Z namespaces/openshift-network-diagnostics/pods/network-check-target-fmxtb/network-check-target-container/network-check-target-container/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.702155875Z namespaces/openshift-network-diagnostics/pods/network-check-target-fmxtb/network-check-target-container/network-check-target-container/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.702192546Z namespaces/openshift-network-diagnostics/pods/network-check-target-kbgr9/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.702218118Z namespaces/openshift-network-diagnostics/pods/network-check-target-kbgr9/network-check-target-kbgr9.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.702296091Z namespaces/openshift-network-diagnostics/pods/network-check-target-kbgr9/network-check-target-container/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.702304092Z namespaces/openshift-network-diagnostics/pods/network-check-target-kbgr9/network-check-target-container/network-check-target-container/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.702308082Z namespaces/openshift-network-diagnostics/pods/network-check-target-kbgr9/network-check-target-container/network-check-target-container/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.702343144Z namespaces/openshift-network-diagnostics/pods/network-check-target-kbgr9/network-check-target-container/network-check-target-container/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.702456319Z namespaces/openshift-network-diagnostics/pods/network-check-target-kbgr9/network-check-target-container/network-check-target-container/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.702524742Z namespaces/openshift-network-diagnostics/pods/network-check-target-kbgr9/network-check-target-container/network-check-target-container/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.702555794Z namespaces/openshift-network-diagnostics/pods/network-check-target-mrh9c/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.702591305Z namespaces/openshift-network-diagnostics/pods/network-check-target-mrh9c/network-check-target-mrh9c.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.702668799Z namespaces/openshift-network-diagnostics/pods/network-check-target-mrh9c/network-check-target-container/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.702678229Z namespaces/openshift-network-diagnostics/pods/network-check-target-mrh9c/network-check-target-container/network-check-target-container/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.702682109Z namespaces/openshift-network-diagnostics/pods/network-check-target-mrh9c/network-check-target-container/network-check-target-container/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.702743192Z namespaces/openshift-network-diagnostics/pods/network-check-target-mrh9c/network-check-target-container/network-check-target-container/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.702830847Z namespaces/openshift-network-diagnostics/pods/network-check-target-mrh9c/network-check-target-container/network-check-target-container/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.702958533Z namespaces/openshift-network-diagnostics/pods/network-check-target-mrh9c/network-check-target-container/network-check-target-container/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.702995574Z namespaces/openshift-network-diagnostics/policy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.703051407Z namespaces/openshift-network-diagnostics/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.70311027Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.703165372Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.703232545Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/rolebindings/network-diagnostics.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.703505688Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/rolebindings/prometheus-k8s.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.703591242Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.703631404Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/roles/network-diagnostics.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.703722158Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/roles/prometheus-k8s.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.703767031Z namespaces/openshift-network-diagnostics/route.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.703796632Z namespaces/openshift-network-diagnostics/route.openshift.io/routes.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.703854085Z namespaces/openshift-network-node-identity/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.703899667Z namespaces/openshift-network-node-identity/openshift-network-node-identity.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.703957049Z namespaces/openshift-network-node-identity/apps.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.704016692Z namespaces/openshift-network-node-identity/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.7041847Z namespaces/openshift-network-node-identity/apps/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.704224932Z namespaces/openshift-network-node-identity/apps/daemonsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.704310176Z namespaces/openshift-network-node-identity/apps/deployments.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.70438729Z namespaces/openshift-network-node-identity/apps/replicasets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.704464563Z namespaces/openshift-network-node-identity/apps/statefulsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.704511175Z namespaces/openshift-network-node-identity/autoscaling/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.704553677Z namespaces/openshift-network-node-identity/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.704585449Z namespaces/openshift-network-node-identity/batch/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.704639532Z namespaces/openshift-network-node-identity/batch/cronjobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.704718955Z namespaces/openshift-network-node-identity/batch/jobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.704764827Z namespaces/openshift-network-node-identity/build.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.704793919Z namespaces/openshift-network-node-identity/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.704887663Z namespaces/openshift-network-node-identity/build.openshift.io/builds.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.704934335Z namespaces/openshift-network-node-identity/coordination.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.704942446Z namespaces/openshift-network-node-identity/coordination.k8s.io/leases/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.704979408Z namespaces/openshift-network-node-identity/coordination.k8s.io/leases/ovnkube-identity.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.705048411Z namespaces/openshift-network-node-identity/core/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.705080142Z namespaces/openshift-network-node-identity/core/configmaps.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.705198878Z namespaces/openshift-network-node-identity/core/endpoints.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.705271531Z namespaces/openshift-network-node-identity/core/events.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.705358735Z namespaces/openshift-network-node-identity/core/persistentvolumeclaims.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.705439119Z namespaces/openshift-network-node-identity/core/pods.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.705518843Z namespaces/openshift-network-node-identity/core/replicationcontrollers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.70566532Z namespaces/openshift-network-node-identity/core/secrets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.705769605Z namespaces/openshift-network-node-identity/core/services.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.705815437Z namespaces/openshift-network-node-identity/core/serviceaccounts/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.705859909Z namespaces/openshift-network-node-identity/core/serviceaccounts/network-node-identity.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.705910051Z namespaces/openshift-network-node-identity/discovery.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.705953923Z namespaces/openshift-network-node-identity/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.705997716Z namespaces/openshift-network-node-identity/image.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.706052808Z namespaces/openshift-network-node-identity/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.706110051Z namespaces/openshift-network-node-identity/k8s.ovn.org/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.706140852Z namespaces/openshift-network-node-identity/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.706232516Z namespaces/openshift-network-node-identity/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.70630917Z namespaces/openshift-network-node-identity/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.706357402Z namespaces/openshift-network-node-identity/monitoring.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.706396484Z namespaces/openshift-network-node-identity/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.706431136Z namespaces/openshift-network-node-identity/networking.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.706483388Z namespaces/openshift-network-node-identity/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.70652877Z namespaces/openshift-network-node-identity/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.706536231Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.706572912Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.706718229Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.706893447Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.707053705Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.707285966Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.707475015Z namespaces/openshift-network-node-identity/policy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.707503616Z namespaces/openshift-network-node-identity/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.707562019Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.70756943Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.707612461Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/rolebindings/network-node-identity-leases.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.707653553Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.707683515Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/roles/network-node-identity-leases.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.707747818Z namespaces/openshift-network-node-identity/route.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.7077893Z namespaces/openshift-network-node-identity/route.openshift.io/routes.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.707833232Z namespaces/openshift-network-operator/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.707866713Z namespaces/openshift-network-operator/openshift-network-operator.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.707936447Z namespaces/openshift-network-operator/apps.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.707976469Z namespaces/openshift-network-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.708025051Z namespaces/openshift-network-operator/apps/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.708083584Z namespaces/openshift-network-operator/apps/daemonsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.708191469Z namespaces/openshift-network-operator/apps/deployments.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.708265082Z namespaces/openshift-network-operator/apps/replicasets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.708346606Z namespaces/openshift-network-operator/apps/statefulsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.708398298Z namespaces/openshift-network-operator/apps/daemonsets/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.7084375Z namespaces/openshift-network-operator/apps/daemonsets/iptables-alerter.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.708503553Z namespaces/openshift-network-operator/autoscaling/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.708541145Z namespaces/openshift-network-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.708588387Z namespaces/openshift-network-operator/batch/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.708617049Z namespaces/openshift-network-operator/batch/cronjobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.708739514Z namespaces/openshift-network-operator/batch/jobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.708786207Z namespaces/openshift-network-operator/build.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.708814618Z namespaces/openshift-network-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.708908562Z namespaces/openshift-network-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.708952824Z namespaces/openshift-network-operator/coordination.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.708961405Z namespaces/openshift-network-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.708994686Z namespaces/openshift-network-operator/coordination.k8s.io/leases/network-operator-lock.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.70906062Z namespaces/openshift-network-operator/core/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.709114932Z namespaces/openshift-network-operator/core/configmaps.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.709441178Z namespaces/openshift-network-operator/core/endpoints.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.709571694Z namespaces/openshift-network-operator/core/events.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.709732561Z namespaces/openshift-network-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.709809835Z namespaces/openshift-network-operator/core/pods.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.709992853Z namespaces/openshift-network-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.710106549Z namespaces/openshift-network-operator/core/secrets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.710228755Z namespaces/openshift-network-operator/core/services.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.710287667Z namespaces/openshift-network-operator/core/configmaps/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.710324629Z namespaces/openshift-network-operator/core/configmaps/applied-cluster.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.710407783Z namespaces/openshift-network-operator/core/configmaps/iptables-alerter-script.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.710470496Z namespaces/openshift-network-operator/core/serviceaccounts/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.710507808Z namespaces/openshift-network-operator/core/serviceaccounts/iptables-alerter.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.71055923Z namespaces/openshift-network-operator/discovery.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.710601172Z namespaces/openshift-network-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.710657985Z namespaces/openshift-network-operator/image.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.710694486Z namespaces/openshift-network-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.710742499Z namespaces/openshift-network-operator/k8s.ovn.org/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.71077618Z namespaces/openshift-network-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.710861554Z namespaces/openshift-network-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.710939808Z namespaces/openshift-network-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.711015832Z namespaces/openshift-network-operator/monitoring.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.711073084Z namespaces/openshift-network-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.711108986Z namespaces/openshift-network-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.711162479Z namespaces/openshift-network-operator/monitoring.coreos.com/prometheusrules/openshift-network-operator-ipsec-rules.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.711222011Z namespaces/openshift-network-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.711255453Z namespaces/openshift-network-operator/monitoring.coreos.com/servicemonitors/network-operator.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.711329956Z namespaces/openshift-network-operator/networking.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.711364698Z namespaces/openshift-network-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.71141515Z namespaces/openshift-network-operator/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.711425231Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.711453442Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.711605589Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.712020189Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.712236639Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.712432578Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.712622127Z namespaces/openshift-network-operator/pods/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.712629297Z namespaces/openshift-network-operator/pods/iptables-alerter-bqvvn/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.712666249Z namespaces/openshift-network-operator/pods/iptables-alerter-bqvvn/iptables-alerter-bqvvn.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.712743013Z namespaces/openshift-network-operator/pods/iptables-alerter-bqvvn/iptables-alerter/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.712750743Z namespaces/openshift-network-operator/pods/iptables-alerter-bqvvn/iptables-alerter/iptables-alerter/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.712759744Z namespaces/openshift-network-operator/pods/iptables-alerter-bqvvn/iptables-alerter/iptables-alerter/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.712788295Z namespaces/openshift-network-operator/pods/iptables-alerter-bqvvn/iptables-alerter/iptables-alerter/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.712877519Z namespaces/openshift-network-operator/pods/iptables-alerter-bqvvn/iptables-alerter/iptables-alerter/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.712945052Z namespaces/openshift-network-operator/pods/iptables-alerter-bqvvn/iptables-alerter/iptables-alerter/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.712980024Z namespaces/openshift-network-operator/pods/iptables-alerter-dvqr7/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.713027126Z namespaces/openshift-network-operator/pods/iptables-alerter-dvqr7/iptables-alerter-dvqr7.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.713268438Z namespaces/openshift-network-operator/pods/iptables-alerter-dvqr7/iptables-alerter/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.713334961Z namespaces/openshift-network-operator/pods/iptables-alerter-dvqr7/iptables-alerter/iptables-alerter/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.713342221Z namespaces/openshift-network-operator/pods/iptables-alerter-dvqr7/iptables-alerter/iptables-alerter/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.713392704Z namespaces/openshift-network-operator/pods/iptables-alerter-dvqr7/iptables-alerter/iptables-alerter/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.713480968Z namespaces/openshift-network-operator/pods/iptables-alerter-dvqr7/iptables-alerter/iptables-alerter/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.713551551Z namespaces/openshift-network-operator/pods/iptables-alerter-dvqr7/iptables-alerter/iptables-alerter/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.713581352Z namespaces/openshift-network-operator/pods/iptables-alerter-r6wkn/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.713611374Z namespaces/openshift-network-operator/pods/iptables-alerter-r6wkn/iptables-alerter-r6wkn.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.713692167Z namespaces/openshift-network-operator/pods/iptables-alerter-r6wkn/iptables-alerter/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.713699848Z namespaces/openshift-network-operator/pods/iptables-alerter-r6wkn/iptables-alerter/iptables-alerter/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.713703748Z namespaces/openshift-network-operator/pods/iptables-alerter-r6wkn/iptables-alerter/iptables-alerter/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.71373765Z namespaces/openshift-network-operator/pods/iptables-alerter-r6wkn/iptables-alerter/iptables-alerter/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.713826314Z namespaces/openshift-network-operator/pods/iptables-alerter-r6wkn/iptables-alerter/iptables-alerter/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.713893357Z namespaces/openshift-network-operator/pods/iptables-alerter-r6wkn/iptables-alerter/iptables-alerter/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.713921868Z namespaces/openshift-network-operator/policy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.71395437Z namespaces/openshift-network-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.714030114Z namespaces/openshift-network-operator/route.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.714063615Z namespaces/openshift-network-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.714117767Z namespaces/openshift-node/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.714125538Z namespaces/openshift-node/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.714129468Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.71416564Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.714314547Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.714492495Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.714634242Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.714837332Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.715178397Z namespaces/openshift-operator-lifecycle-manager/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.715216019Z namespaces/openshift-operator-lifecycle-manager/openshift-operator-lifecycle-manager.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.715313874Z namespaces/openshift-operator-lifecycle-manager/apps.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.715343945Z namespaces/openshift-operator-lifecycle-manager/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.715396428Z namespaces/openshift-operator-lifecycle-manager/apps/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.715429389Z namespaces/openshift-operator-lifecycle-manager/apps/daemonsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.715599327Z namespaces/openshift-operator-lifecycle-manager/apps/deployments.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.715782016Z namespaces/openshift-operator-lifecycle-manager/apps/replicasets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.715991496Z namespaces/openshift-operator-lifecycle-manager/apps/statefulsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.71608078Z namespaces/openshift-operator-lifecycle-manager/autoscaling/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.716226187Z namespaces/openshift-operator-lifecycle-manager/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.716269409Z namespaces/openshift-operator-lifecycle-manager/batch/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.716343562Z namespaces/openshift-operator-lifecycle-manager/batch/cronjobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.716614655Z namespaces/openshift-operator-lifecycle-manager/batch/jobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.716880208Z namespaces/openshift-operator-lifecycle-manager/build.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.716908759Z namespaces/openshift-operator-lifecycle-manager/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.717135209Z namespaces/openshift-operator-lifecycle-manager/build.openshift.io/builds.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.717255255Z namespaces/openshift-operator-lifecycle-manager/core/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.71735134Z namespaces/openshift-operator-lifecycle-manager/core/configmaps.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.717546329Z namespaces/openshift-operator-lifecycle-manager/core/endpoints.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.717631093Z namespaces/openshift-operator-lifecycle-manager/core/events.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.717822472Z namespaces/openshift-operator-lifecycle-manager/core/persistentvolumeclaims.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.71799934Z namespaces/openshift-operator-lifecycle-manager/core/pods.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.718337136Z namespaces/openshift-operator-lifecycle-manager/core/replicationcontrollers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.718461502Z namespaces/openshift-operator-lifecycle-manager/core/secrets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.718925104Z namespaces/openshift-operator-lifecycle-manager/core/services.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.71906726Z namespaces/openshift-operator-lifecycle-manager/discovery.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.719154444Z namespaces/openshift-operator-lifecycle-manager/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.719259849Z namespaces/openshift-operator-lifecycle-manager/image.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.719405586Z namespaces/openshift-operator-lifecycle-manager/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.71948016Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.719553593Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.719745142Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.719829926Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.719891309Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.719938341Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.719981703Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.720041466Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/prometheusrules/olm-alert-rules.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.720151101Z namespaces/openshift-operator-lifecycle-manager/networking.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.720250726Z namespaces/openshift-operator-lifecycle-manager/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.720458106Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.720472826Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.720733389Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.720914477Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.721444742Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.721708525Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.722024159Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.722311313Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.722379336Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/operatorgroups/olm-operators.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.722475151Z namespaces/openshift-operator-lifecycle-manager/policy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.722502642Z namespaces/openshift-operator-lifecycle-manager/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.722596306Z namespaces/openshift-operator-lifecycle-manager/route.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.722692821Z namespaces/openshift-operator-lifecycle-manager/route.openshift.io/routes.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.722776215Z namespaces/openshift-operators/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.722785725Z namespaces/openshift-operators/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.722790806Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.722852129Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.723001096Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.723421675Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.723562162Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.72394201Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.724321408Z namespaces/openshift-operators/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.724354099Z namespaces/openshift-operators/operators.coreos.com/operatorgroups/global-operators.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.724409992Z namespaces/openshift-ovn-kubernetes/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.724491075Z namespaces/openshift-ovn-kubernetes/openshift-ovn-kubernetes.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.724639333Z namespaces/openshift-ovn-kubernetes/apps.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.724739317Z namespaces/openshift-ovn-kubernetes/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.724784099Z namespaces/openshift-ovn-kubernetes/apps/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.724821941Z namespaces/openshift-ovn-kubernetes/apps/daemonsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.72523305Z namespaces/openshift-ovn-kubernetes/apps/deployments.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.725344656Z namespaces/openshift-ovn-kubernetes/apps/replicasets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.725498603Z namespaces/openshift-ovn-kubernetes/apps/statefulsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.725603068Z namespaces/openshift-ovn-kubernetes/apps/daemonsets/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.72564946Z namespaces/openshift-ovn-kubernetes/apps/daemonsets/ovnkube-node.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.725785147Z namespaces/openshift-ovn-kubernetes/autoscaling/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.725881801Z namespaces/openshift-ovn-kubernetes/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.725988566Z namespaces/openshift-ovn-kubernetes/batch/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.726103131Z namespaces/openshift-ovn-kubernetes/batch/cronjobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.726249748Z namespaces/openshift-ovn-kubernetes/batch/jobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.726329332Z namespaces/openshift-ovn-kubernetes/build.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.726365174Z namespaces/openshift-ovn-kubernetes/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.726547822Z namespaces/openshift-ovn-kubernetes/build.openshift.io/builds.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.726593945Z namespaces/openshift-ovn-kubernetes/coordination.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.726603735Z namespaces/openshift-ovn-kubernetes/coordination.k8s.io/leases/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.726633466Z namespaces/openshift-ovn-kubernetes/coordination.k8s.io/leases/ovn-kubernetes-master.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.726688009Z namespaces/openshift-ovn-kubernetes/core/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.72672254Z namespaces/openshift-ovn-kubernetes/core/configmaps.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.72693054Z namespaces/openshift-ovn-kubernetes/core/endpoints.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.727018905Z namespaces/openshift-ovn-kubernetes/core/events.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.72756195Z namespaces/openshift-ovn-kubernetes/core/persistentvolumeclaims.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.727680436Z namespaces/openshift-ovn-kubernetes/core/pods.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.728256383Z namespaces/openshift-ovn-kubernetes/core/replicationcontrollers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.728333727Z namespaces/openshift-ovn-kubernetes/core/secrets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.728498434Z namespaces/openshift-ovn-kubernetes/core/services.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.728554867Z namespaces/openshift-ovn-kubernetes/core/configmaps/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.728632461Z namespaces/openshift-ovn-kubernetes/core/configmaps/ovnkube-config.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.728770767Z namespaces/openshift-ovn-kubernetes/core/configmaps/ovnkube-script-lib.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.728934895Z namespaces/openshift-ovn-kubernetes/core/serviceaccounts/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.728969056Z namespaces/openshift-ovn-kubernetes/core/serviceaccounts/ovn-kubernetes-control-plane.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.729084092Z namespaces/openshift-ovn-kubernetes/core/serviceaccounts/ovn-kubernetes-node.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.729120364Z namespaces/openshift-ovn-kubernetes/core/services/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.729172186Z namespaces/openshift-ovn-kubernetes/core/services/ovn-kubernetes-node.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.729226028Z namespaces/openshift-ovn-kubernetes/discovery.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.729272111Z namespaces/openshift-ovn-kubernetes/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.729325263Z namespaces/openshift-ovn-kubernetes/image.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.729359975Z namespaces/openshift-ovn-kubernetes/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.729412697Z namespaces/openshift-ovn-kubernetes/k8s.cni.cncf.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.729422007Z namespaces/openshift-ovn-kubernetes/k8s.cni.cncf.io/network-attachment-definitions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.729451479Z namespaces/openshift-ovn-kubernetes/k8s.cni.cncf.io/network-attachment-definitions/default.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.729497571Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.729529683Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.729647928Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.729806746Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.729943912Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.730022656Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.73011145Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.730155742Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/prometheusrules/networking-rules.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.730239476Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.730281378Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/servicemonitors/monitor-ovn-node.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.73033427Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.730341591Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/operatorpkis/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.730376552Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/operatorpkis/ovn.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.730465467Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/operatorpkis/signer.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.730513539Z namespaces/openshift-ovn-kubernetes/networking.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.730606113Z namespaces/openshift-ovn-kubernetes/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.730704218Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.730712818Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.730792122Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.730952139Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.731158929Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.731305206Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.731509656Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.731699635Z namespaces/openshift-ovn-kubernetes/pods/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.731710025Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bwrjz/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.731773328Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bwrjz/ovnkube-node-bwrjz.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.731941146Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bwrjz/kube-rbac-proxy-node/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.731951486Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bwrjz/kube-rbac-proxy-node/kube-rbac-proxy-node/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.731956677Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bwrjz/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.731981188Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bwrjz/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.732119425Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bwrjz/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.732192828Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bwrjz/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.73222951Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bwrjz/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.73223724Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bwrjz/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.732246351Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bwrjz/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.732253001Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bwrjz/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.732377847Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bwrjz/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.73244598Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bwrjz/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.732479151Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bwrjz/nbdb/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.732483932Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bwrjz/nbdb/nbdb/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.732487942Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bwrjz/nbdb/nbdb/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.732525864Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bwrjz/nbdb/nbdb/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.732645889Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bwrjz/nbdb/nbdb/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.732714013Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bwrjz/nbdb/nbdb/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.732742954Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bwrjz/northd/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.732747574Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bwrjz/northd/northd/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.732752734Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bwrjz/northd/northd/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.732778545Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bwrjz/northd/northd/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.732894251Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bwrjz/northd/northd/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.732957754Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bwrjz/northd/northd/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.732996716Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bwrjz/ovn-acl-logging/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.733022317Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bwrjz/ovn-acl-logging/ovn-acl-logging/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.733032397Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bwrjz/ovn-acl-logging/ovn-acl-logging/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.733042618Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bwrjz/ovn-acl-logging/ovn-acl-logging/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.733146053Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bwrjz/ovn-acl-logging/ovn-acl-logging/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.733219926Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bwrjz/ovn-acl-logging/ovn-acl-logging/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.733254188Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bwrjz/ovn-controller/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.733261158Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bwrjz/ovn-controller/ovn-controller/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.733265068Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bwrjz/ovn-controller/ovn-controller/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.73329837Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bwrjz/ovn-controller/ovn-controller/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.733588074Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bwrjz/ovn-controller/ovn-controller/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.733760232Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bwrjz/ovn-controller/ovn-controller/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.733827255Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bwrjz/ovnkube-controller/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.733834645Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bwrjz/ovnkube-controller/ovnkube-controller/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.734025604Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bwrjz/ovnkube-controller/ovnkube-controller/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.734116068Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bwrjz/ovnkube-controller/ovnkube-controller/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.73606466Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bwrjz/ovnkube-controller/ovnkube-controller/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.736135294Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bwrjz/ovnkube-controller/ovnkube-controller/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.736154334Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bwrjz/sbdb/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.736163645Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bwrjz/sbdb/sbdb/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.736173265Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bwrjz/sbdb/sbdb/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.736219117Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bwrjz/sbdb/sbdb/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.736321182Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bwrjz/sbdb/sbdb/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.736382915Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bwrjz/sbdb/sbdb/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.736419317Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jkg9m/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.73649079Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jkg9m/ovnkube-node-jkg9m.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.736637947Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jkg9m/kube-rbac-proxy-node/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.736645367Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jkg9m/kube-rbac-proxy-node/kube-rbac-proxy-node/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.736649508Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jkg9m/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.736684129Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jkg9m/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.736816155Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jkg9m/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.736884439Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jkg9m/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.73691172Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jkg9m/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.7369162Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jkg9m/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.73692012Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jkg9m/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.736968043Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jkg9m/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.73713693Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jkg9m/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.737228895Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jkg9m/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.737262516Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jkg9m/nbdb/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.737270787Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jkg9m/nbdb/nbdb/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.737274757Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jkg9m/nbdb/nbdb/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.737309769Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jkg9m/nbdb/nbdb/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.737406233Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jkg9m/nbdb/nbdb/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.737476257Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jkg9m/nbdb/nbdb/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.737513668Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jkg9m/northd/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.737520179Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jkg9m/northd/northd/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.737528689Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jkg9m/northd/northd/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.737536369Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jkg9m/northd/northd/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.737680906Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jkg9m/northd/northd/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.737748849Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jkg9m/northd/northd/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.737788981Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jkg9m/ovn-acl-logging/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.737796892Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jkg9m/ovn-acl-logging/ovn-acl-logging/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.737801222Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jkg9m/ovn-acl-logging/ovn-acl-logging/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.737824983Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jkg9m/ovn-acl-logging/ovn-acl-logging/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.737918007Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jkg9m/ovn-acl-logging/ovn-acl-logging/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.73798615Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jkg9m/ovn-acl-logging/ovn-acl-logging/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.738030423Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jkg9m/ovn-controller/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.738039623Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jkg9m/ovn-controller/ovn-controller/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.738043563Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jkg9m/ovn-controller/ovn-controller/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.738086755Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jkg9m/ovn-controller/ovn-controller/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.738411001Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jkg9m/ovn-controller/ovn-controller/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.738524866Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jkg9m/ovn-controller/ovn-controller/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.738552377Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jkg9m/ovnkube-controller/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.738559068Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jkg9m/ovnkube-controller/ovnkube-controller/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.738564318Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jkg9m/ovnkube-controller/ovnkube-controller/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.738604259Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jkg9m/ovnkube-controller/ovnkube-controller/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.740543631Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jkg9m/ovnkube-controller/ovnkube-controller/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.740637515Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jkg9m/ovnkube-controller/ovnkube-controller/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.740657336Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jkg9m/sbdb/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.740670657Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jkg9m/sbdb/sbdb/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.740675087Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jkg9m/sbdb/sbdb/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.740718679Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jkg9m/sbdb/sbdb/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.740826044Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jkg9m/sbdb/sbdb/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.740891317Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jkg9m/sbdb/sbdb/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.740934509Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-tpvc8/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.740996622Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-tpvc8/ovnkube-node-tpvc8.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.74116582Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-tpvc8/kube-rbac-proxy-node/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.741176321Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-tpvc8/kube-rbac-proxy-node/kube-rbac-proxy-node/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.741180321Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-tpvc8/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.741214953Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-tpvc8/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.741342608Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-tpvc8/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.741407731Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-tpvc8/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.741451133Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-tpvc8/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.741458604Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-tpvc8/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.741462724Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-tpvc8/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.741492226Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-tpvc8/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.741635892Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-tpvc8/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.741703245Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-tpvc8/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.741744127Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-tpvc8/nbdb/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.741751578Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-tpvc8/nbdb/nbdb/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.741755588Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-tpvc8/nbdb/nbdb/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.741788089Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-tpvc8/nbdb/nbdb/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.741885454Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-tpvc8/nbdb/nbdb/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.741957537Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-tpvc8/nbdb/nbdb/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.742055532Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-tpvc8/northd/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.742071983Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-tpvc8/northd/northd/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.742078663Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-tpvc8/northd/northd/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.742089434Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-tpvc8/northd/northd/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.742200469Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-tpvc8/northd/northd/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.742271132Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-tpvc8/northd/northd/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.742288263Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-tpvc8/ovn-acl-logging/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.742300973Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-tpvc8/ovn-acl-logging/ovn-acl-logging/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.742305044Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-tpvc8/ovn-acl-logging/ovn-acl-logging/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.742348086Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-tpvc8/ovn-acl-logging/ovn-acl-logging/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.742466691Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-tpvc8/ovn-acl-logging/ovn-acl-logging/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.742531184Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-tpvc8/ovn-acl-logging/ovn-acl-logging/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.742576017Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-tpvc8/ovn-controller/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.742585737Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-tpvc8/ovn-controller/ovn-controller/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.742589957Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-tpvc8/ovn-controller/ovn-controller/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.742613258Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-tpvc8/ovn-controller/ovn-controller/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.742908312Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-tpvc8/ovn-controller/ovn-controller/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.742976995Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-tpvc8/ovn-controller/ovn-controller/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.743023507Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-tpvc8/ovnkube-controller/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.743034288Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-tpvc8/ovnkube-controller/ovnkube-controller/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.74308219Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-tpvc8/ovnkube-controller/ovnkube-controller/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.743133653Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-tpvc8/ovnkube-controller/ovnkube-controller/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.74498349Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-tpvc8/ovnkube-controller/ovnkube-controller/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.745077154Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-tpvc8/ovnkube-controller/ovnkube-controller/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.745099885Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-tpvc8/sbdb/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.745107516Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-tpvc8/sbdb/sbdb/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.745115026Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-tpvc8/sbdb/sbdb/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.745128286Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-tpvc8/sbdb/sbdb/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.745276853Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-tpvc8/sbdb/sbdb/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.745346667Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-tpvc8/sbdb/sbdb/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.745365458Z namespaces/openshift-ovn-kubernetes/policy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.745422Z namespaces/openshift-ovn-kubernetes/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.745475723Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.745484923Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.745498814Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.745608509Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/openshift-ovn-kubernetes-nodes-identity-limited.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.745856231Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/prometheus-k8s.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.745905433Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.745948765Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.746061961Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/openshift-ovn-kubernetes-node-limited.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.746165805Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/prometheus-k8s.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.746201637Z namespaces/openshift-ovn-kubernetes/route.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.74625511Z namespaces/openshift-ovn-kubernetes/route.openshift.io/routes.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.746289751Z namespaces/openshift-route-controller-manager/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.746296372Z namespaces/openshift-route-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.746305602Z namespaces/openshift-route-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.746354154Z namespaces/openshift-route-controller-manager/coordination.k8s.io/leases/openshift-route-controllers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.746405377Z namespaces/openshift-route-controller-manager/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.746414057Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.746428078Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.746636777Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.746845027Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.747000605Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.747233916Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.747519659Z namespaces/openshift-service-ca-operator/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.747554991Z namespaces/openshift-service-ca-operator/openshift-service-ca-operator.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.747631314Z namespaces/openshift-service-ca-operator/apps.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.747666086Z namespaces/openshift-service-ca-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.747702058Z namespaces/openshift-service-ca-operator/apps/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.74775839Z namespaces/openshift-service-ca-operator/apps/daemonsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.747836304Z namespaces/openshift-service-ca-operator/apps/deployments.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.747948749Z namespaces/openshift-service-ca-operator/apps/replicasets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.748068995Z namespaces/openshift-service-ca-operator/apps/statefulsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.748109337Z namespaces/openshift-service-ca-operator/autoscaling/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.748157009Z namespaces/openshift-service-ca-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.748201211Z namespaces/openshift-service-ca-operator/batch/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.748238913Z namespaces/openshift-service-ca-operator/batch/cronjobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.748323787Z namespaces/openshift-service-ca-operator/batch/jobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.748370659Z namespaces/openshift-service-ca-operator/build.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.74840374Z namespaces/openshift-service-ca-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.748492265Z namespaces/openshift-service-ca-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.748539697Z namespaces/openshift-service-ca-operator/coordination.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.748548837Z namespaces/openshift-service-ca-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.748586119Z namespaces/openshift-service-ca-operator/coordination.k8s.io/leases/service-ca-operator-lock.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.748633301Z namespaces/openshift-service-ca-operator/core/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.748670183Z namespaces/openshift-service-ca-operator/core/configmaps.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.748786569Z namespaces/openshift-service-ca-operator/core/endpoints.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.748893294Z namespaces/openshift-service-ca-operator/core/events.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.749126034Z namespaces/openshift-service-ca-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.749264071Z namespaces/openshift-service-ca-operator/core/pods.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.749366236Z namespaces/openshift-service-ca-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.749493982Z namespaces/openshift-service-ca-operator/core/secrets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.749700642Z namespaces/openshift-service-ca-operator/core/services.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.749760744Z namespaces/openshift-service-ca-operator/discovery.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.749793876Z namespaces/openshift-service-ca-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.749857029Z namespaces/openshift-service-ca-operator/image.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.749903571Z namespaces/openshift-service-ca-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.749946393Z namespaces/openshift-service-ca-operator/k8s.ovn.org/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.749978915Z namespaces/openshift-service-ca-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.75009247Z namespaces/openshift-service-ca-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.750170084Z namespaces/openshift-service-ca-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.750211026Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.750259638Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.75031393Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.750343842Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/servicemonitors/service-ca-operator.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.750401845Z namespaces/openshift-service-ca-operator/networking.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.750446437Z namespaces/openshift-service-ca-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.750491409Z namespaces/openshift-service-ca-operator/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.750497909Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.750542521Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.750690048Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.750867067Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.751016824Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.751284296Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.751506587Z namespaces/openshift-service-ca-operator/pods/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.751516197Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d6c75f84f-b7nxx/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.751540108Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d6c75f84f-b7nxx/service-ca-operator-5d6c75f84f-b7nxx.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.751630212Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d6c75f84f-b7nxx/service-ca-operator/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.751638703Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d6c75f84f-b7nxx/service-ca-operator/service-ca-operator/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.751642763Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d6c75f84f-b7nxx/service-ca-operator/service-ca-operator/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.751673225Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d6c75f84f-b7nxx/service-ca-operator/service-ca-operator/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.751829682Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d6c75f84f-b7nxx/service-ca-operator/service-ca-operator/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.751985909Z namespaces/openshift-service-ca-operator/policy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.752043332Z namespaces/openshift-service-ca-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.752098765Z namespaces/openshift-service-ca-operator/route.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.752126706Z namespaces/openshift-service-ca-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.752182709Z namespaces/openshift-service-ca/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.752228551Z namespaces/openshift-service-ca/openshift-service-ca.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.752301384Z namespaces/openshift-service-ca/apps.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.752319375Z namespaces/openshift-service-ca/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.752378858Z namespaces/openshift-service-ca/apps/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.75242404Z namespaces/openshift-service-ca/apps/daemonsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.752506294Z namespaces/openshift-service-ca/apps/deployments.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.752606778Z namespaces/openshift-service-ca/apps/replicasets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.752738804Z namespaces/openshift-service-ca/apps/statefulsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.752758365Z namespaces/openshift-service-ca/autoscaling/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.752831479Z namespaces/openshift-service-ca/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.752867631Z namespaces/openshift-service-ca/batch/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.752908663Z namespaces/openshift-service-ca/batch/cronjobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.752988596Z namespaces/openshift-service-ca/batch/jobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.753053959Z namespaces/openshift-service-ca/build.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.753101812Z namespaces/openshift-service-ca/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.753180435Z namespaces/openshift-service-ca/build.openshift.io/builds.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.753223437Z namespaces/openshift-service-ca/coordination.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.753232108Z namespaces/openshift-service-ca/coordination.k8s.io/leases/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.75327444Z namespaces/openshift-service-ca/coordination.k8s.io/leases/service-ca-controller-lock.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.753320812Z namespaces/openshift-service-ca/core/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.753367794Z namespaces/openshift-service-ca/core/configmaps.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.753470059Z namespaces/openshift-service-ca/core/endpoints.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.753548823Z namespaces/openshift-service-ca/core/events.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.753661628Z namespaces/openshift-service-ca/core/persistentvolumeclaims.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.753739892Z namespaces/openshift-service-ca/core/pods.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.753839646Z namespaces/openshift-service-ca/core/replicationcontrollers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.753968082Z namespaces/openshift-service-ca/core/secrets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.754160322Z namespaces/openshift-service-ca/core/services.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.754210444Z namespaces/openshift-service-ca/discovery.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.754252396Z namespaces/openshift-service-ca/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.754300988Z namespaces/openshift-service-ca/image.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.7543423Z namespaces/openshift-service-ca/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.754381762Z namespaces/openshift-service-ca/k8s.ovn.org/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.754502158Z namespaces/openshift-service-ca/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.754584362Z namespaces/openshift-service-ca/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.754668445Z namespaces/openshift-service-ca/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.754702827Z namespaces/openshift-service-ca/monitoring.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.754753849Z namespaces/openshift-service-ca/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.754803442Z namespaces/openshift-service-ca/networking.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.754840603Z namespaces/openshift-service-ca/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.754879545Z namespaces/openshift-service-ca/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.754886056Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.754932518Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.755099826Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.755293905Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.755424181Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.75562518Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.755805629Z namespaces/openshift-service-ca/pods/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.755813829Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-9w82r/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.755850201Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-9w82r/service-ca-658d7b7c78-9w82r.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.755927145Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-9w82r/service-ca-controller/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.755934575Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-9w82r/service-ca-controller/service-ca-controller/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.755938555Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-9w82r/service-ca-controller/service-ca-controller/logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.755972607Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-9w82r/service-ca-controller/service-ca-controller/logs/current.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.756300032Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-9w82r/service-ca-controller/service-ca-controller/logs/previous.insecure.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.756447129Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-9w82r/service-ca-controller/service-ca-controller/logs/previous.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.7564744Z namespaces/openshift-service-ca/policy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.756522903Z namespaces/openshift-service-ca/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.756574495Z namespaces/openshift-service-ca/route.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.756613087Z namespaces/openshift-service-ca/route.openshift.io/routes.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.756652109Z namespaces/openshift-user-workload-monitoring/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.756707381Z namespaces/openshift-user-workload-monitoring/openshift-user-workload-monitoring.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.756761884Z namespaces/openshift-user-workload-monitoring/apps.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.756794166Z namespaces/openshift-user-workload-monitoring/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.756847188Z namespaces/openshift-user-workload-monitoring/apps/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.756879919Z namespaces/openshift-user-workload-monitoring/apps/daemonsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.756999935Z namespaces/openshift-user-workload-monitoring/apps/deployments.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.75710147Z namespaces/openshift-user-workload-monitoring/apps/replicasets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.757186314Z namespaces/openshift-user-workload-monitoring/apps/statefulsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.757234236Z namespaces/openshift-user-workload-monitoring/autoscaling/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.757268028Z namespaces/openshift-user-workload-monitoring/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.757300929Z namespaces/openshift-user-workload-monitoring/batch/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.757339861Z namespaces/openshift-user-workload-monitoring/batch/cronjobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.757433085Z namespaces/openshift-user-workload-monitoring/batch/jobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.757471547Z namespaces/openshift-user-workload-monitoring/build.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.757499909Z namespaces/openshift-user-workload-monitoring/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.757592083Z namespaces/openshift-user-workload-monitoring/build.openshift.io/builds.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.757621544Z namespaces/openshift-user-workload-monitoring/core/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.757672557Z namespaces/openshift-user-workload-monitoring/core/configmaps.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.757776112Z namespaces/openshift-user-workload-monitoring/core/endpoints.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.757853995Z namespaces/openshift-user-workload-monitoring/core/events.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.757988942Z namespaces/openshift-user-workload-monitoring/core/persistentvolumeclaims.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.758100077Z namespaces/openshift-user-workload-monitoring/core/pods.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.758183941Z namespaces/openshift-user-workload-monitoring/core/replicationcontrollers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.758312867Z namespaces/openshift-user-workload-monitoring/core/secrets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.758472484Z namespaces/openshift-user-workload-monitoring/core/services.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.758519527Z namespaces/openshift-user-workload-monitoring/discovery.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.758571459Z namespaces/openshift-user-workload-monitoring/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.758612831Z namespaces/openshift-user-workload-monitoring/image.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.758646953Z namespaces/openshift-user-workload-monitoring/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.758697515Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.758726246Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.758816661Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.759187408Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.75922833Z namespaces/openshift-user-workload-monitoring/monitoring.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.759285533Z namespaces/openshift-user-workload-monitoring/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.759326245Z namespaces/openshift-user-workload-monitoring/networking.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.759382877Z namespaces/openshift-user-workload-monitoring/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.759413599Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.759418269Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.759471061Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.759619748Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.759804507Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.759947194Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.760182465Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.760354883Z namespaces/openshift-user-workload-monitoring/policy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.760386584Z namespaces/openshift-user-workload-monitoring/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.760444617Z namespaces/openshift-user-workload-monitoring/route.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.760490669Z namespaces/openshift-user-workload-monitoring/route.openshift.io/routes.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.760522351Z namespaces/openshift/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.760579453Z namespaces/openshift/openshift.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.760627156Z namespaces/openshift/apps.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.760659377Z namespaces/openshift/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.76072477Z namespaces/openshift/apps/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.760766192Z namespaces/openshift/apps/daemonsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.760846126Z namespaces/openshift/apps/deployments.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.76092517Z namespaces/openshift/apps/replicasets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.761022784Z namespaces/openshift/apps/statefulsets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.761066067Z namespaces/openshift/autoscaling/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.761103518Z namespaces/openshift/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.76114521Z namespaces/openshift/batch/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.761186102Z namespaces/openshift/batch/cronjobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.761269306Z namespaces/openshift/batch/jobs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.761299127Z namespaces/openshift/build.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.76135843Z namespaces/openshift/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.761430404Z namespaces/openshift/build.openshift.io/builds.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.761475346Z namespaces/openshift/core/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.761503227Z namespaces/openshift/core/configmaps.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.761611262Z namespaces/openshift/core/endpoints.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.761689106Z namespaces/openshift/core/events.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.76177038Z namespaces/openshift/core/persistentvolumeclaims.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.761850973Z namespaces/openshift/core/pods.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.761927247Z namespaces/openshift/core/replicationcontrollers.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.762069394Z namespaces/openshift/core/secrets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.762232141Z namespaces/openshift/core/services.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.762318415Z namespaces/openshift/discovery.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.762365228Z namespaces/openshift/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.76241286Z namespaces/openshift/image.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.762439821Z namespaces/openshift/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.763859168Z namespaces/openshift/image.openshift.io/imagestreams/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.76390482Z namespaces/openshift/image.openshift.io/imagestreams/cli-artifacts.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.763995904Z namespaces/openshift/image.openshift.io/imagestreams/cli.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.764109499Z namespaces/openshift/image.openshift.io/imagestreams/dotnet-runtime.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.764243386Z namespaces/openshift/image.openshift.io/imagestreams/dotnet.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.764348611Z namespaces/openshift/image.openshift.io/imagestreams/driver-toolkit.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.764435015Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-eap-openshift-java11.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.764592122Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-eap-openshift.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.76474944Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-java-openshift.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.764880126Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-java11-openshift.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.76497633Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-karaf-openshift-jdk11.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.765102156Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-karaf-openshift.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.765238433Z namespaces/openshift/image.openshift.io/imagestreams/golang.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.765343548Z namespaces/openshift/image.openshift.io/imagestreams/httpd.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.765451853Z namespaces/openshift/image.openshift.io/imagestreams/installer-artifacts.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.765541477Z namespaces/openshift/image.openshift.io/imagestreams/installer.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.765634401Z namespaces/openshift/image.openshift.io/imagestreams/java-runtime.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.765737186Z namespaces/openshift/image.openshift.io/imagestreams/java.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.765855722Z namespaces/openshift/image.openshift.io/imagestreams/jboss-datagrid73-openshift.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.765971787Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp3-openjdk11-openshift.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.766093953Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp3-openjdk11-runtime-openshift.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.766200768Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp4-openjdk11-openshift.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.766339884Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp4-openjdk11-runtime-openshift.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.766440019Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk11-openshift.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.766539204Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk11-runtime-openshift.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.766639558Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk8-openshift.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.766736443Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk8-runtime-openshift.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.766835088Z namespaces/openshift/image.openshift.io/imagestreams/jboss-webserver57-openjdk11-tomcat9-openshift-ubi8.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.766945683Z namespaces/openshift/image.openshift.io/imagestreams/jboss-webserver57-openjdk8-tomcat9-openshift-ubi8.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.767140132Z namespaces/openshift/image.openshift.io/imagestreams/jenkins-agent-base.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.767244157Z namespaces/openshift/image.openshift.io/imagestreams/jenkins.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.767367763Z namespaces/openshift/image.openshift.io/imagestreams/mariadb.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.767474628Z namespaces/openshift/image.openshift.io/imagestreams/must-gather.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.767561642Z namespaces/openshift/image.openshift.io/imagestreams/mysql.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.767674957Z namespaces/openshift/image.openshift.io/imagestreams/network-tools.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.767760931Z namespaces/openshift/image.openshift.io/imagestreams/nginx.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.767888517Z namespaces/openshift/image.openshift.io/imagestreams/nodejs.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.768064066Z namespaces/openshift/image.openshift.io/imagestreams/oauth-proxy.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.76816384Z namespaces/openshift/image.openshift.io/imagestreams/openjdk-11-rhel7.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.768276716Z namespaces/openshift/image.openshift.io/imagestreams/perl.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.76837704Z namespaces/openshift/image.openshift.io/imagestreams/php.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.768489486Z namespaces/openshift/image.openshift.io/imagestreams/postgresql.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.768619082Z namespaces/openshift/image.openshift.io/imagestreams/postgresql13-for-sso75-openshift-rhel8.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.768713446Z namespaces/openshift/image.openshift.io/imagestreams/postgresql13-for-sso76-openshift-rhel8.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.76880157Z namespaces/openshift/image.openshift.io/imagestreams/python.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.768936747Z namespaces/openshift/image.openshift.io/imagestreams/redhat-openjdk18-openshift.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.769094554Z namespaces/openshift/image.openshift.io/imagestreams/redis.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.76920983Z namespaces/openshift/image.openshift.io/imagestreams/ruby.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.769315384Z namespaces/openshift/image.openshift.io/imagestreams/sso75-openshift-rhel8.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.769415629Z namespaces/openshift/image.openshift.io/imagestreams/sso76-openshift-rhel8.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.769545495Z namespaces/openshift/image.openshift.io/imagestreams/tests.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.76963764Z namespaces/openshift/image.openshift.io/imagestreams/tools.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.769779076Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-11-runtime.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.769911072Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-11.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.770054359Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-17-runtime.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.770171045Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-17.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.77029373Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-21-runtime.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.770390825Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-21.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.77048548Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-8-runtime.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.770609865Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-8.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.770692849Z namespaces/openshift/k8s.ovn.org/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.770739902Z namespaces/openshift/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.770845736Z namespaces/openshift/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.770942451Z namespaces/openshift/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.770961142Z namespaces/openshift/monitoring.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.771029985Z namespaces/openshift/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.771097158Z namespaces/openshift/networking.k8s.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.771151831Z namespaces/openshift/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.771183432Z namespaces/openshift/operators.coreos.com/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.771189413Z namespaces/openshift/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.771231755Z namespaces/openshift/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.771485307Z namespaces/openshift/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.771720648Z namespaces/openshift/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.771865395Z namespaces/openshift/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.772098775Z namespaces/openshift/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.772313625Z namespaces/openshift/policy/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.772364278Z namespaces/openshift/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.77241641Z namespaces/openshift/route.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.772428851Z namespaces/openshift/route.openshift.io/routes.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.772488764Z namespaces/openshift/template.openshift.io/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.772497594Z namespaces/openshift/template.openshift.io/templates/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.772542336Z namespaces/openshift/template.openshift.io/templates/cache-service.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.772661192Z namespaces/openshift/template.openshift.io/templates/cakephp-mysql-example.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.772787258Z namespaces/openshift/template.openshift.io/templates/cakephp-mysql-persistent.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.772899053Z namespaces/openshift/template.openshift.io/templates/dancer-mysql-example.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.773026319Z namespaces/openshift/template.openshift.io/templates/dancer-mysql-persistent.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.773208148Z namespaces/openshift/template.openshift.io/templates/datagrid-service.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.773316533Z namespaces/openshift/template.openshift.io/templates/django-psql-example.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.773431098Z namespaces/openshift/template.openshift.io/templates/django-psql-persistent.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.773540933Z namespaces/openshift/template.openshift.io/templates/eap-xp3-basic-s2i.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.77367488Z namespaces/openshift/template.openshift.io/templates/eap-xp4-basic-s2i.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.773910161Z namespaces/openshift/template.openshift.io/templates/eap74-basic-s2i.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.774057468Z namespaces/openshift/template.openshift.io/templates/eap74-https-s2i.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.774236396Z namespaces/openshift/template.openshift.io/templates/eap74-sso-s2i.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.774381623Z namespaces/openshift/template.openshift.io/templates/httpd-example.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.774487918Z namespaces/openshift/template.openshift.io/templates/jenkins-ephemeral-monitored.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.774626594Z namespaces/openshift/template.openshift.io/templates/jenkins-ephemeral.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.774734199Z namespaces/openshift/template.openshift.io/templates/jenkins-persistent-monitored.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.774847645Z namespaces/openshift/template.openshift.io/templates/jenkins-persistent.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.77496865Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk11-tomcat9-ubi8-basic-s2i.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.775198021Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk11-tomcat9-ubi8-https-s2i.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.775325557Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk8-tomcat9-ubi8-basic-s2i.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.775448403Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk8-tomcat9-ubi8-https-s2i.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.775560298Z namespaces/openshift/template.openshift.io/templates/mariadb-ephemeral.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.775671574Z namespaces/openshift/template.openshift.io/templates/mariadb-persistent.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.775773848Z namespaces/openshift/template.openshift.io/templates/mysql-ephemeral.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.775876533Z namespaces/openshift/template.openshift.io/templates/mysql-persistent.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.775979198Z namespaces/openshift/template.openshift.io/templates/nginx-example.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.776123605Z namespaces/openshift/template.openshift.io/templates/nodejs-postgresql-example.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.776242421Z namespaces/openshift/template.openshift.io/templates/nodejs-postgresql-persistent.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.776349326Z namespaces/openshift/template.openshift.io/templates/openjdk-web-basic-s2i.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.77645078Z namespaces/openshift/template.openshift.io/templates/postgresql-ephemeral.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.776554465Z namespaces/openshift/template.openshift.io/templates/postgresql-persistent.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.77665698Z namespaces/openshift/template.openshift.io/templates/rails-pgsql-persistent.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.776781656Z namespaces/openshift/template.openshift.io/templates/rails-postgresql-example.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.776897241Z namespaces/openshift/template.openshift.io/templates/react-web-app-example.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.777018477Z namespaces/openshift/template.openshift.io/templates/redis-ephemeral.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.777118912Z namespaces/openshift/template.openshift.io/templates/redis-persistent.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.777227327Z namespaces/openshift/template.openshift.io/templates/s2i-fuse712-spring-boot-2-camel-rest-3scale.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.777376744Z namespaces/openshift/template.openshift.io/templates/s2i-fuse712-spring-boot-2-camel-xml.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.777486759Z namespaces/openshift/template.openshift.io/templates/s2i-fuse712-spring-boot-2-camel.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.777591654Z namespaces/openshift/template.openshift.io/templates/sso75-https.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.77772024Z namespaces/openshift/template.openshift.io/templates/sso75-ocp4-x509-https.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.777848136Z namespaces/openshift/template.openshift.io/templates/sso75-ocp4-x509-postgresql-persistent.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.777974932Z namespaces/openshift/template.openshift.io/templates/sso75-postgresql-persistent.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.77815632Z namespaces/openshift/template.openshift.io/templates/sso75-postgresql.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.778282446Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-https.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.778435664Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-postgresql-persistent.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.77856628Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-postgresql.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.778696856Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-x509-https.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.778820662Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-x509-postgresql-persistent.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.778896615Z network_logs/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.778937427Z network_logs/cluster_scale [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.779023951Z network_logs/ippools.whereabouts.cni.cncf.io [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.779121236Z network_logs/multi-networkpolicy [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.779198169Z network_logs/net-attach-def [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.779293894Z network_logs/overlappingrangeipreservations.whereabouts.cni.cncf.io [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.779442191Z network_logs/ovn_kubernetes_top_pods [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.779534415Z network_logs/ovnk_database_store.tar.gz [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.782162619Z nodes/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.782204311Z nodes/debug [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.782303946Z nodes/ip-10-0-129-80.ec2.internal/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.782373459Z nodes/ip-10-0-129-80.ec2.internal/cpu_affinities.json [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.782690634Z nodes/ip-10-0-129-80.ec2.internal/dmesg [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.782975207Z nodes/ip-10-0-129-80.ec2.internal/ethtool_channels [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.783146825Z nodes/ip-10-0-129-80.ec2.internal/ethtool_features [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.78323962Z nodes/ip-10-0-129-80.ec2.internal/ip-10-0-129-80.ec2.internal_logs_kubelet.gz [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.783827787Z nodes/ip-10-0-129-80.ec2.internal/irq_affinities.json [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.783968854Z nodes/ip-10-0-129-80.ec2.internal/lscpu [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.784112871Z nodes/ip-10-0-129-80.ec2.internal/lspci [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.784208185Z nodes/ip-10-0-129-80.ec2.internal/podresources.json [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.78431797Z nodes/ip-10-0-129-80.ec2.internal/pods_info.json [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.784437406Z nodes/ip-10-0-129-80.ec2.internal/proc_cmdline [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.78451984Z nodes/ip-10-0-129-80.ec2.internal/sysinfo.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.786299963Z nodes/ip-10-0-129-80.ec2.internal/sysinfo.tgz [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.786465201Z nodes/ip-10-0-133-86.ec2.internal/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.786484262Z nodes/ip-10-0-133-86.ec2.internal/cpu_affinities.json [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.786831648Z nodes/ip-10-0-133-86.ec2.internal/dmesg [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.787117282Z nodes/ip-10-0-133-86.ec2.internal/ethtool_channels [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.787195116Z nodes/ip-10-0-133-86.ec2.internal/ethtool_features [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.78728848Z nodes/ip-10-0-133-86.ec2.internal/ip-10-0-133-86.ec2.internal_logs_kubelet.gz [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.787791664Z nodes/ip-10-0-133-86.ec2.internal/irq_affinities.json [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.787933771Z nodes/ip-10-0-133-86.ec2.internal/lscpu [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.788088548Z nodes/ip-10-0-133-86.ec2.internal/lspci [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.788176332Z nodes/ip-10-0-133-86.ec2.internal/podresources.json [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.788285137Z nodes/ip-10-0-133-86.ec2.internal/pods_info.json [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.788403823Z nodes/ip-10-0-133-86.ec2.internal/proc_cmdline [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.788492587Z nodes/ip-10-0-133-86.ec2.internal/sysinfo.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.790235499Z nodes/ip-10-0-133-86.ec2.internal/sysinfo.tgz [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.790396986Z nodes/ip-10-0-140-240.ec2.internal/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.790426258Z nodes/ip-10-0-140-240.ec2.internal/cpu_affinities.json [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.790711601Z nodes/ip-10-0-140-240.ec2.internal/dmesg [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.790985624Z nodes/ip-10-0-140-240.ec2.internal/ethtool_channels [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.791142311Z nodes/ip-10-0-140-240.ec2.internal/ethtool_features [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.791307169Z nodes/ip-10-0-140-240.ec2.internal/ip-10-0-140-240.ec2.internal_logs_kubelet.gz [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.79174061Z nodes/ip-10-0-140-240.ec2.internal/irq_affinities.json [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.791889026Z nodes/ip-10-0-140-240.ec2.internal/lscpu [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.792034113Z nodes/ip-10-0-140-240.ec2.internal/lspci [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.792125458Z nodes/ip-10-0-140-240.ec2.internal/podresources.json [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.792227953Z nodes/ip-10-0-140-240.ec2.internal/pods_info.json [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.792343838Z nodes/ip-10-0-140-240.ec2.internal/proc_cmdline [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.792425922Z nodes/ip-10-0-140-240.ec2.internal/sysinfo.log [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.79429898Z nodes/ip-10-0-140-240.ec2.internal/sysinfo.tgz [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.794484329Z pod_network_connectivity_check/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.79452281Z pod_network_connectivity_check/podnetworkconnectivitychecks.yaml [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.794631156Z static-pods/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.794641076Z static-pods/kube-apiserver/ [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.802839502Z [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.802862453Z sent 72,345 bytes received 8,692,717 bytes 5,843,374.67 bytes/sec [must-gather] [must-gather-cwdzq] OUT 2026-07-29T12:13:39.802873563Z total size is 110,479,543 speedup is 12.60 [must-gather] [must-gather ] OUT 2026-07-29T12:13:40.120729868Z namespace/openshift-must-gather-jjmjm deleted [must-gather] [must-gather] [must-gather] Reprinting Cluster State: [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: 7e1b384b-0e13-46c5-87f6-afaf6fa56fd7 [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.26" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [git-push-artifacts] WORK_DIR: /workspace/odh-ci-artifacts [git-push-artifacts] REPO_PATH: opendatahub-io/odh-build-metadata [git-push-artifacts] REPO_BRANCH: ci-artifacts [git-push-artifacts] SPARSE_FILE_PATH: test-artifacts/docs [git-push-artifacts] SOURCE_PATH: /workspace/artifacts-dir [git-push-artifacts] DEST_PATH: test-artifacts/kserve-group-test-5wppp [git-push-artifacts] ALWAYS_PASS: false [git-push-artifacts] configuring gh token [git-push-artifacts] taking github token from Konflux bot [git-push-artifacts] Initialized empty Git repository in /workspace/odh-ci-artifacts/.git/ [git-push-artifacts] Using partial fetch with sparse checkout for: test-artifacts/docs [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] * [new branch] ci-artifacts -> origin/ci-artifacts [git-push-artifacts] Already on 'ci-artifacts' [git-push-artifacts] branch 'ci-artifacts' set up to track 'origin/ci-artifacts'. [git-push-artifacts] TASK_NAME=kserve-group-test-5wppp-e2e-llm-inference-service [git-push-artifacts] PIPELINERUN_NAME=kserve-group-test-5wppp [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] Already up to date. [git-push-artifacts] -rw-r--r--. 1 root 1001540000 17080032 Jul 29 12:15 /workspace/odh-ci-artifacts/test-artifacts/kserve-group-test-5wppp/e2e-llm-inference-service.tar.gz [git-push-artifacts] [ci-artifacts 9a4657a] Updating CI Artifacts in e2e-llm-inference-service [git-push-artifacts] 1 file changed, 0 insertions(+), 0 deletions(-) [git-push-artifacts] create mode 100644 test-artifacts/kserve-group-test-5wppp/e2e-llm-inference-service.tar.gz [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] Already up to date. [git-push-artifacts] To https://github.com/opendatahub-io/odh-build-metadata.git [git-push-artifacts] 8b9eb7d..9a4657a ci-artifacts -> ci-artifacts [fail-if-needed] Failing pipeline because deploy-and-e2e step failed container step-fail-if-needed has failed : [{"key":"StartedAt","value":"2026-07-29T12:15:37.817Z","type":3}]